1.1 异构并行计算

1.1 Heterogeneous Parallel Computing

自 2003 年以来，半导体行业已经确定了微处理器设计的两条主要轨迹[Hwu2008]。多核轨迹旨在在进入多核的同时保持顺序程序的执行速度。多核始于两个核心处理器，核心数量随着每一代半导体工艺而增加。当前的一个例子是最新的 Intel Core i7 ™微处理器，具有四个处理器内核，每个处理器内核都存在乱序、多指令问题处理器实现完整的 X86 指令集，支持具有两个硬件线程的超线程，旨在最大限度地提高顺序程序的执行速度。相比之下，多线程轨迹更关注并行应用程序的执行吞吐量。多线程从大量线程开始，并且线程数量再次随着每一代的增加而增加。当前的一个范例是 NVIDIA GTX680 图形处理单元 (GPU)，具有 16,384 个线程，在大量简单、有序的管道中执行。

Since 2003, the semiconductor industry has settled on two main trajectories for designing microprocessors [Hwu2008]. The multicore trajectory seeks to maintain the execution speed of sequential programs while moving into multiple cores. The multicores began with two core processors with the number of cores increasing with each semiconductor process generation. A current exemplar is the recent Intel Core i7™ microprocessor with four processor cores, each of which is an out-of-order, multiple instruction issue processor implementing the full X86 instruction set, supporting hyperthreading with two hardware threads, designed to maximize the execution speed of sequential programs. In contrast, the many-thread trajectory focuses more on the execution throughput of parallel applications. The many-threads began with a large number of threads, and once again, the number of threads increases with each generation. A current exemplar is the NVIDIA GTX680 graphics processing unit (GPU) with 16,384 threads, executing in a large number of simple, in-order pipelines.

自2003年以来，多线程处理器，尤其是GPU，一直在浮点性能竞赛中处于领先地位。截至2012年，多线程GPU和多核CPU之间的峰值浮点计算吞吐量之比约为10。这些并不一定应用程序速度，但仅仅是执行资源在这些芯片中可能支持的原始速度：2012 年为 1.5 teraflops 与 150 gigaflops 双精度。

Many-threads processors, especially the GPUs, have led the race of floating-point performance since 2003. As of 2012, the ratio of peak floating-point calculation throughput between many-thread GPUs and multicore CPUs is about 10. These are not necessarily application speeds, but are merely the raw speed that the execution resources can potentially support in these chips: 1.5 teraflops versus 150 gigaflops double precision in 2012.

并行执行和顺序执行之间如此巨大的性能差距已经形成了显着的“电势”积累，在某些时候，必须做出一些让步。现在我们已经达到了这一点。迄今为止，这种巨大的性能差距已经促使许多应用程序开发人员将其软件的计算密集型部分转移到 GPU 上执行。毫不奇怪，这些计算密集型部分也是并行编程的主要目标——当有更多工作要做时，就有更多机会在协作的并行工作人员之间分配工作。

Such a large performance gap between parallel and sequential execution has amounted to a significant “electrical potential” build-up, and at some point, something will have to give. We have reached that point now. To date, this large performance gap has already motivated many application developers to move the computationally intensive parts of their software to GPUs for execution. Not surprisingly, these computationally intensive parts are also the prime target of parallel programming—when there is more work to do, there is more opportunity to divide the work among cooperating parallel workers.

有人可能会问，为什么多线程 GPU 和通用多核 CPU 之间的峰值性能差距如此之大。答案在于两种类型处理器之间基本设计理念的差异，如图1.1所示。 CPU 的设计针对顺序代码性能进行了优化。它利用复杂的控制逻辑，允许来自单个线程的指令并行执行，甚至不按顺序执行，同时保持顺序执行的外观。更重要的是，提供大容量高速缓冲存储器以减少大型复杂应用的指令和数据访问延迟。控制逻辑和高速缓冲存储器都不会影响峰值计算速度。截至 2012 年，高端通用多核微处理器通常具有六到八个大型处理器内核和数兆字节的片上高速缓存，旨在提供强大的顺序代码性能。

One might ask why there is such a large peak-performance gap between many-threads GPUs and general-purpose multicore CPUs. The answer lies in the differences in the fundamental design philosophies between the two types of processors, as illustrated in Figure 1.1. The design of a CPU is optimized for sequential code performance. It makes use of sophisticated control logic to allow instructions from a single thread to execute in parallel or even out of their sequential order while maintaining the appearance of sequential execution. More importantly, large cache memories are provided to reduce the instruction and data access latencies of large complex applications. Neither control logic nor cache memories contribute to the peak calculation speed. As of 2012, the high-end general-purpose multicore microprocessors typically have six to eight large processor cores and multiple megabytes of on-chip cache memories designed to deliver strong sequential code performance.

内存带宽是另一个重要问题。许多应用程序的速度受到从设备传送数据的速率的限制。内存系统进入处理器。图形芯片的运行内存带宽大约是同时可用的 CPU 芯片的六倍。 2006 年底，GeForce 8800 GTX（或简称为 G80）能够以每秒约 85 GB (GB/s) 的速度将数据移入和移出其主动态随机存取存储器 (DRAM)，因为图形帧缓冲区要求和宽松的内存模型（各种系统软件、应用程序和输入/输出 (I/O) 设备期望其内存访问的工作方式）。最新的 GTX680 芯片支持大约 200 GB/s。相比之下，通用处理器必须满足传统操作系统、应用程序和 I/O 设备的要求，这使得内存带宽更难以增加。因此，CPU 在内存带宽方面将在一段时间内继续处于劣势。

Memory bandwidth is another important issue. The speed of many applications is limited by the rate at which data can be delivered from the memory system into the processors. Graphics chips have been operating at approximately six times the memory bandwidth of contemporaneously available CPU chips. In late 2006, GeForce 8800 GTX, or simply G80, was capable of moving data at about 85 gigabytes per second (GB/s) in and out of its main dynamic random-access memory (DRAM) because of graphics frame buffer requirements and the relaxed memory model (the way various system software, applications, and input/output (I/O) devices expect how their memory accesses work). The more recent GTX680 chip supports about 200 GB/s. In contrast, general-purpose processors have to satisfy requirements from legacy operating systems, applications, and I/O devices that make memory bandwidth more difficult to increase. As a result, CPUs will continue to be at a disadvantage in terms of memory bandwidth for some time.

GPU 的设计理念是由快速发展的视频游戏行业塑造的，该行业对高级游戏中每个视频帧执行大量浮点计算的能力施加了巨大的经济压力。这种需求促使 GPU 供应商寻找方法来最大化专用于浮点计算的芯片面积和功率预算。普遍的解决方案是针对大量线程的执行吞吐量进行优化。该设计允许流水线内存通道和算术运算具有较长的延迟，从而节省芯片面积和功耗。内存访问硬件和算术单元的面积和功率的减少允许设计人员在芯片上拥有更多的硬件和算术单元，从而提高总执行吞吐量。

The design philosophy of GPUs is shaped by the fast-growing video game industry that exerts tremendous economic pressure for the ability to perform a massive number of floating-point calculations per video frame in advanced games. This demand motivates GPU vendors to look for ways to maximize the chip area and power budget dedicated to floating-point calculations. The prevailing solution is to optimize for the execution throughput of massive numbers of threads. The design saves chip area and power by allowing pipelined memory channels and arithmetic operations to have long latency. The reduced area and power of the memory access hardware and arithmetic units allows the designers to have more of them on a chip and thus increase the total execution throughput.

应用软件预计将使用大量并行线程来编写。硬件充分利用了大量的当其中一些线程正在等待长延迟内存访问或算术运算时，线程可以找到要做的工作。提供小型高速缓冲存储器来帮助控制这些应用程序的带宽要求，以便访问相同存储器数据的多个线程不需要全部转到DRAM。这种设计风格通常称为面向吞吐量的设计，因为它力求最大化大量线程的总执行吞吐量，同时允许单个线程可能需要更长的执行时间。

The application software is expected to be written with a large number of parallel threads. The hardware takes advantage of the large number of threads to find work to do when some of them are waiting for long-latency memory accesses or arithmetic operations. Small cache memories are provided to help control the bandwidth requirements of these applications so that multiple threads that access the same memory data do not need to all go to the DRAM. This design style is commonly referred to as throughput-oriented design since it strives to maximize the total execution throughput of a large number of threads while allowing individual threads to take a potentially much longer time to execute.

另一方面，CPU 的设计目的是最大限度地减少单个线程的执行延迟。大型末级片上缓存旨在捕获频繁访问的数据，并将一些长延迟内存访问转换为短延迟缓存访问。算术单元和操作数数据传送逻辑还被设计成以增加芯片面积和功率的使用为代价来最小化操作的有效延迟。通过减少同一线程内操作的延迟，CPU 硬件可以减少每个单独线程的执行延迟。然而，大型高速缓冲存储器、低延迟算术单元和复杂的操作数传送逻辑消耗了芯片面积和功率，而这些面积和功率本来可以用于提供更多算术执行单元和存储器访问通道。这种设计风格通常称为面向延迟的设计。

The CPUs, on the other hand, are designed to minimize the execution latency of a single thread. Large last-level on-chip caches are designed to capture frequently accessed data and convert some of the long-latency memory accesses into short-latency cache accesses. The arithmetic units and operand data delivery logic are also designed to minimize the effective latency of operation at the cost of increased use of chip area and power. By reducing the latency of operations within the same thread, the CPU hardware reduces the execution latency of each individual thread. However, the large cache memory, low-latency arithmetic units, and sophisticated operand delivery logic consume chip area and power that could be otherwise used to provide more arithmetic execution units and memory access channels. This design style is commonly referred to as latency-oriented design.

现在应该很清楚，GPU 被设计为并行的、面向吞吐量的计算引擎，它们在某些 CPU 设计为能很好执行的任务上表现不佳。对于只有一个或很少线程的程序，具有较低操作延迟的CPU可以获得比GPU高得多的性能。当程序有大量线程时，具有较高执行吞吐量的GPU可以获得比CPU高得多的性能。因此，我们应该预料到许多应用程序会同时使用 CPU 和 GPU，在 CPU 上执行顺序部分，在 GPU 上执行数值密集型部分。这就是 NVIDIA 于 2007 年推出的 CUDA 编程模型旨在支持应用程序的 CPU-GPU 联合执行的原因。¹支持 CPU-GPU 联合执行的需求进一步反映在最近的编程模型中，例如 OpenCL（参见第 14 章）、OpenACC（参见第 15 章）和 C++AMP（参见第 18 章）。

It should be clear now that GPUs are designed as parallel, throughput-oriented computing engines and they will not perform well on some tasks on which CPUs are designed to perform well. For programs that have one or very few threads, CPUs with lower operation latencies can achieve much higher performance than GPUs. When a program has a large number of threads, GPUs with higher execution throughput can achieve much higher performance than CPUs. Therefore, one should expect that many applications use both CPUs and GPUs, executing the sequential parts on the CPU and numerically intensive parts on the GPUs. This is why the CUDA programming model, introduced by NVIDIA in 2007, is designed to support joint CPU–GPU execution of an application.¹ The demand for supporting joint CPU–GPU execution is further reflected in more recent programming models such as OpenCL (see Chapter 14), OpenACC (see Chapter 15), and C++AMP (see Chapter 18).

同样重要的是要注意，当应用程序开发人员选择运行其处理器时，性能并不是唯一的决定因素。应用程序。其他几个因素可能更为重要。首先也是最重要的是，所选处理器必须在市场上占有很大的份额，称为处理器的安装基础。原因很简单。大量的客户群最能证明软件开发成本的合理性。在市场份额较小的处理器上运行的应用程序不会有大量的客户群。这一直是传统并行计算系统的一个主要问题，与通用微处理器相比，传统并行计算系统的市场份额可以忽略不计。只有少数由政府和大公司资助的精英应用程序在这些传统的并行计算系统上成功开发。随着多核 GPU 的出现，这种情况发生了变化。由于 GPU 在 PC 市场的受欢迎，其销量已达数亿。几乎所有 PC 都配有 GPU。迄今为止，已有超过 4 亿个支持 CUDA 的 GPU 正在使用中。这是大规模并行计算首次在大众市场产品中变得可行。如此庞大的市场占有率使这些 GPU 对应用程序开发人员来说具有经济吸引力。

It is also important to note that performance is not the only decision factor when application developers choose the processors for running their applications. Several other factors can be even more important. First and foremost, the processors of choice must have a very large presence in the marketplace, referred to as the installed base of the processor. The reason is very simple. The cost of software development is best justified by a very large customer population. Applications that run on a processor with a small market presence will not have a large customer base. This has been a major problem with traditional parallel computing systems that have negligible market presence compared to general-purpose microprocessors. Only a few elite applications funded by government and large corporations have been successfully developed on these traditional parallel computing systems. This has changed with many-core GPUs. Due to their popularity in the PC market, GPUs have been sold by the hundreds of millions. Virtually all PCs have GPUs in them. There are more than 400 million CUDA-enabled GPUs in use to date. This is the first time that massively parallel computing is feasible with a mass-market product. Such a large market presence has made these GPUs economically attractive targets for application developers.

另一个重要的决定因素是实用的外形因素和易于访问性。直到 2006 年，并行软件应用程序通常运行在数据中心服务器或部门集群上。但这样的执行环境往往会限制这些应用程序的使用。比如在医学影像这样的应用中，发表一篇基于64节点集群机的论文就可以了。但磁共振成像 (MRI) 机器的实际临床应用是基于 PC 和特殊硬件加速器的某种组合。原因很简单，GE 和西门子等制造商无法将带有计算服务器机架的 MRI 出售到临床环境中，而这在学术部门环境中很常见。事实上，美国国立卫生研究院（NIH）有一段时间拒绝资助并行编程项目：他们认为并行软件的影响将受到限制，因为基于集群的大型机器无法在临床环境中工作。如今，GE 推出配备 GPU 的 MRI 产品，NIH 资助使用 GPU 计算的研究。

Another important decision factor is practical form factors and easy accessibility. Until 2006, parallel software applications usually ran on data center servers or departmental clusters. But such execution environments tend to limit the use of these applications. For example, in an application such as medical imaging, it is fine to publish a paper based on a 64-node cluster machine. But actual clinical applications on magnetic resonance imaging (MRI) machines have been based on some combination of a PC and special hardware accelerators. The simple reason is that manufacturers such as GE and Siemens cannot sell MRIs with racks of compute server boxes into clinical settings, while this is common in academic departmental settings. In fact, National Institutes of Health (NIH) refused to fund parallel programming projects for some time: they felt that the impact of parallel software would be limited because huge cluster-based machines would not work in the clinical setting. Today, GE ships MRI products with GPUs and NIH funds research using GPU computing.

选择用于执行数值计算应用程序的处理器时的另一个重要考虑因素是对电气和电子工程师协会 (IEEE) 浮点标准的支持级别。该标准使得跨不同供应商的处理器获得可预测的结果成为可能。虽然早期 GPU 对 IEEE 浮点标准的支持并不强，但自 G80 推出以来，新一代 GPU 的情况也发生了变化。正如我们将要讨论的在第 7 章中，GPU 对 IEEE 浮点标准的支持已经与 CPU 的支持相当。因此，可以预期更多的数值应用程序将被移植到 GPU，并产生与 CPU 相当的结果值。到 2009 年，一个主要的遗留问题是 GPU 的浮点运算单元主要是单精度的。真正需要双精度浮点运算单元的应用程序并不适合 GPU 执行。然而，随着最近的 GPU 的出现，这种情况发生了变化，双精度执行速度接近单精度的一半左右，这是高端 CPU 内核达到的水平。这使得 GPU 适合更多的数值应用。

Yet another important consideration in selecting a processor for executing numeric computing applications is the level of support for the Institute of Electrical and Electronic Engineers’ (IEEE) floating-point standard. The standard makes it possible to have predictable results across processors from different vendors. While the support for the IEEE floating-point standard was not strong in early GPUs, this has also changed for new generations of GPUs since the introduction of the G80. As we will discuss in Chapter 7, GPU support for the IEEE floating-point standard has become comparable with that of the CPUs. As a result, one can expect that more numerical applications will be ported to GPUs and yield comparable result values as the CPUs. Up to 2009, a major remaining issue was that the GPUs’ floating-point arithmetic units were primarily single precision. Applications that truly require double-precision floating-point arithmetic units were not suitable for GPU execution. However, this has changed with the recent GPUs of which the double-precision execution speed approaches about half of that of single precision, a level that high-end CPU cores achieve. This makes the GPUs suitable for even more numerical applications.

直到 2006 年，图形芯片都非常难以使用，因为程序员必须使用相当于图形 API（应用程序编程接口）功能来访问处理器内核，这意味着需要 OpenGL 或 Direct3D 技术来对这些芯片进行编程。更简单地说，计算必须表示为以某种方式绘制像素的函数，以便在这些早期 GPU 上执行。该技术称为 GPGPU（使用图形处理单元的通用编程）。即使使用更高级别的编程环境，底层代码仍然需要适合用于绘制像素的 API。这些 API 限制了人们可以为早期 GPGPU 实际编写的应用程序类型。因此，它并没有成为一种广泛的编程现象。尽管如此，这项技术还是足够令人兴奋，激发了一些英勇的努力和出色的研究成果。

Until 2006, graphics chips were very difficult to use because programmers had to use the equivalent of graphics API (application programming interface) functions to access the processor cores, meaning that OpenGL or Direct3D techniques were needed to program these chips. Stated more simply, a computation must be expressed as a function that paints a pixel in some way to execute on these early GPUs. This technique was called GPGPU (general-purpose programming using a graphics processing unit). Even with a higher-level programming environment, the underlying code still needs to fit into the APIs that are designed to paint pixels. These APIs limit the kinds of applications that one can actually write for early GPGPUs. Consequently, it did not become a widespread programming phenomenon. Nonetheless, this technology was sufficiently exciting to inspire some heroic efforts and excellent research results.

但随着 2007 年 CUDA [NVIDIA2007]的发布，一切都发生了变化。 NVIDIA 开始在其 GPU 芯片上投入硅片区域，以方便并行编程。这不仅仅代表软件的变化；芯片中添加了额外的硬件。在 G80 及其后续并行计算芯片中，CUDA 程序根本不再通过图形接口。相反，硅芯片上的新通用并行编程接口可以满足 CUDA 程序的要求。通用编程接口极大地扩展了可以轻松为 GPU 开发的应用程序类型。此外，所有其他软件层也都进行了重做，以便程序员可以使用熟悉的C/C++编程工具。我们的一些学生尝试使用旧的基于 OpenGL 的编程接口来完成实验室作业，他们的经验帮助他们极大地欣赏了计算应用程序不再需要使用图形 API 的改进。

But everything changed in 2007 with the release of CUDA [NVIDIA2007]. NVIDIA stared to devote silicon areas on their GPU chips to facilitate the ease of parallel programming.This did not represent software changes alone; additional hardware was added to the chips. In the G80 and its successor chips for parallel computing, CUDA programs no longer go through the graphics interface at all. Instead, a new general-purpose parallel programming interface on the silicon chip serves the requests of CUDA programs. The general-purpose programming interface greatly expands the types of applications that one can easily develop for GPUs. Moreover, all the other software layers were redone as well, so that the programmers can use the familiar C/C++ programming tools. Some of our students tried to do their lab assignments using the old OpenGL-based programming interface, and their experience helped them to greatly appreciate the improvements that eliminated the need for using the graphics APIs for computing applications.

1.2 现代GPU的架构

1.2 Architecture of a Modern GPU

图 1.2显示了典型的支持 CUDA 的 GPU 的架构。它被组织成一系列高度线程化的流式多处理器（SM）。在图 1.3中，两个 SM 形成一个构建块。然而，构建块中的 SM 数量可能因一代 CUDA GPU 的不同而异。此外，在图 1.3中，每个 SM 都有多个共享控制逻辑和指令缓存的流处理器 (SP)。目前，每个 GPU 都配备了数 GB 的图形双倍数据速率 (GDDR) DRAM，在图 1.3中称为全局内存。这些 GDDR DRAM 与 CPU 主板上的系统 DRAM 不同，因为它们本质上是用于图形的帧缓冲存储器。对于图形应用程序，它们保存用于 3D 渲染的视频图像和纹理信息。但对于计算而言，它们充当非常高带宽的片外存储器，尽管延迟比典型的系统存储器稍长。对于大规模并行应用程序，更高的带宽可以弥补更长的延迟。

Figure 1.2 shows the architecture of a typical CUDA-capable GPU. It is organized into an array of highly threaded streaming multiprocessors (SMs). In Figure 1.3, two SMs form a building block. However, the number of SMs in a building block can vary from one generation of CUDA GPUs to another generation. Also, in Figure 1.3, each SM has a number of streaming processors (SPs) that share control logic and an instruction cache. Each GPU currently comes with multiple gigabytes of Graphic Double Data Rate (GDDR) DRAM, referred to as global memory in Figure 1.3. These GDDR DRAMs differ from the system DRAMs on the CPU motherboard in that they are essentially the frame buffer memory that is used for graphics. For graphics applications, they hold video images and texture information for 3D rendering. But for computing, they function as very high bandwidth off-chip memory, though with somewhat longer latency than typical system memory. For massively parallel applications, the higher bandwidth makes up for the longer latency.

G80 引入了 CUDA 架构，拥有 86.4 GB/s 的内存带宽，以及通过 PCI-Express 第 2 代 (Gen2) 接口与 CPU 核心逻辑的通信链路。通过 PCI-E Gen2，CUDA 应用程序可以以 4 GB/s 的速度将数据从系统内存传输到全局内存，同时以 4 GB/s 的速度将数据上传回系统内存。总共有 8 GB/s。最新的 GPU 使用 PCI-E Gen3，每个方向支持 8 GB/s。随着 GPU 内存大小的增长，应用程序越来越多地将数据保存在全局内存中，只有在需要使用仅在 CPU 上可用的库时才偶尔使用 PCI-E 与 CPU 系统内存进行通信。未来通信带宽也有望随着系统内存的CPU总线带宽的增长而增长。

The G80 introduced the CUDA architecture and had 86.4 GB/s of memory bandwidth, plus a communication link to the CPU core logic over a PCI-Express Generation 2 (Gen2) interface. Over PCI-E Gen2, a CUDA application can transfer data from the system memory to the global memory at 4 GB/s, and at the same time upload data back to the system memory at 4 GB/s. Altogether, there is a combined total of 8 GB/s. More recent GPUs use PCI-E Gen3, which supports 8 GB/s in each direction. As the size of GPU memory grows, applications increasingly keep their data in the global memory and only occasionally use the PCI-E to communicate with the CPU system memory if there is need for using a library that is only available on the CPUs. The communication bandwidth is also expected to grow as the CPU bus bandwidth of the system memory grows in the future.

GTX680 拥有 16,384 个线程，双精度超过 1.5 teraflops。一个好的应用程序通常会在该芯片上同时运行 5,000-12,000 个线程。对于习惯 CPU 多线程的用户，请注意，Intel CPU 每个核心支持两个或四个线程，具体取决于机器型号。然而，CPU 越来越多地使用 SIMD（单指令、多数据）指令来实现高数值性能。 GPU硬件和CPU硬件支持的并行度正在快速提高。因此，在开发计算应用程序时争取高水平的并行性非常重要。

With 16,384 threads, the GTX680 exceeds 1.5 teraflops in double precision. A good application typically runs 5,000–12,000 threads simultaneously on this chip. For those who are used to multithreading in CPUs, note that Intel CPUs support two or four threads, depending on the machine model, per core. CPUs, however, are increasingly used with SIMD (single instruction, multiple data) instructions for high numerical performance. The level of parallelism supported by both GPU hardware and CPU hardware is increasing quickly. It is therefore very important to strive for high levels of parallelism when developing computing applications.

1.3 为什么需要更快的速度或并行性？

1.3 Why More Speed or Parallelism?

正如我们在1.1 节中所述，大规模并行编程的主要动机是让应用程序在未来的硬件世代中享受持续的速度提升。人们可能会问为什么应用程序将继续要求提高速度。我们今天拥有的许多应用程序似乎运行得足够快。正如我们将在案例研究章节中讨论的那样，当应用程序适合并行执行时，GPU 上的良好实现可以比单个 CPU 内核上的顺序执行实现超过 100 倍 (100×) 的加速。如果应用程序包含我们所说的数据并行性，那么只需几个小时的工作即可实现 10 倍的加速通常是一项简单的任务。对于除此之外的任何内容，我们邀请您继续阅读！

As we stated in Section 1.1, the main motivation for massively parallel programming is for applications to enjoy continued speed increase in future hardware generations. One might ask why applications will continue to demand increased speed. Many applications that we have today seem to be running quite fast enough. As we will discuss in the case study chapters, when an application is suitable for parallel execution, a good implementation on a GPU can achieve more than 100 times (100×) speedup over sequential execution on a single CPU core. If the application includes what we call data parallelism, it’s often a simple task to achieve a 10× speedup with just a few hours of work. For anything beyond that, we invite you to keep reading!

尽管当今世界有无数的计算应用程序，但未来许多令人兴奋的大众市场应用程序都是我们目前所认为的“超级计算应用程序”或超级应用程序。例如，生物学研究界正在越来越多地进入分子水平。显微镜可以说是分子生物学中最重要的仪器，过去依赖光学或电子仪器。但我们使用这些仪器进行的分子水平观察存在局限性。这些限制可以有效地通过结合计算模型来模拟具有传统仪器设置的边界条件的基础分子活动来解决这一问题。通过模拟，我们可以测量更多的细节并测试更多的假设，这比仅使用传统仪器所能想象的要多。在可预见的未来，就可建模的生物系统的规模以及可在可容忍响应时间内模拟的反应时间长度而言，这些模拟将继续受益于计算速度的提高。这些增强将对科学和医学产生巨大影响。

Despite the myriad of computing applications in today’s world, many exciting mass-market applications of the future are what we currently consider “supercomputing applications,” or super-applications. For example, the biology research community is moving more and more into the molecular level. Microscopes, arguably the most important instrument in molecular biology, used to rely on optics or electronic instrumentation. But there are limitations to the molecular-level observations that we can make with these instruments. These limitations can be effectively addressed by incorporating a computational model to simulate the underlying molecular activities with boundary conditions set by traditional instrumentation. With simulation we can measure even more details and test more hypotheses than can ever be imagined with traditional instrumentation alone. These simulations will continue to benefit from the increasing computing speed in the foreseeable future in terms of the size of the biological system that can be modeled and the length of reaction time that can be simulated within a tolerable response time. These enhancements will have tremendous implications to science and medicine.

对于视频和音频编码和操作等应用，请考虑我们对数字高清 (HD) 电视与老式 NTSC 电视的满意度。一旦我们体验了高清电视的细节水平，就很难再回到旧技术了。但请考虑一下高清电视所需的所有处理。这是一个非常并行的过程，3D 成像和可视化也是如此。未来，视图合成和低分辨率视频的高分辨率显示等新功能将需要电视具有更高的计算能力。在消费者层面，我们将开始拥有越来越多的视频和图像处理应用程序，这些应用程序可以改善图片和视频的焦点、照明和其他关键方面。

For applications such as video and audio coding and manipulation, consider our satisfaction with digital high-definition (HD) TV verses older NTSC TV. Once we experience the level of details in an HDTV, it is very hard to go back to older technology. But consider all the processing that’s needed for that HDTV. It is a very parallel process, as are 3D imaging and visualization. In the future, new functionalities such as view synthesis and high-resolution display of low-resolution videos will demand more computing power in the TV. At the consumer level, we will begin to have an increasing number of video and image processing applications that improve the focus, lighting, and other key aspects of the pictures and videos.

更高的计算速度带来的好处之一是更好的用户界面。考虑一下 Apple 的 iPhone™ 界面：尽管 iPhone 的窗口尺寸仍然有限，但与其他手机设备相比，用户可以享受到更加自然的触摸屏界面。毫无疑问，这些设备的未来版本将采用更高清的 3D 视角、结合虚拟和物理空间信息以增强可用性的应用程序以及基于语音和计算机视觉的界面，从而需要更高的计算速度。

Among the benefits offered by more computing speed are much better user interfaces. Consider Apple’s iPhone™ interfaces: the user enjoys a much more natural interface with the touchscreen than other cell phone devices even though the iPhone still has a limited-size window. Undoubtedly, future versions of these devices will incorporate higher-definition, 3D perspectives, applications that combine virtual and physical space information for enhanced usability, and voice-based and computer vision–based interfaces, requiring even more computing speed.

消费电子游戏领域也正在进行类似的发展。过去，在游戏中驾驶汽车实际上只是一组预先安排好的场景。如果你的车撞到了障碍物，你的车的路线不会改变，只有游戏分数会改变。你的车轮没有弯曲或损坏，驾驶不再困难，无论你是否撞到了车轮，甚至丢了一个车轮。随着计算速度的提高，游戏可以基于动态模拟而不是预先安排的场景。我们可以期待在未来看到更多这样的现实效果：事故会损坏你的车轮，你的在线驾驶体验将更加真实。众所周知，物理效应的真实建模和模拟需要非常大量的计算能力。

Similar developments are underway in consumer electronic gaming. In the past, driving a car in a game was in fact simply a prearranged set of scenes. If your car bumped into an obstacle, the course of your vehicle did not change, only the game score changed. Your wheels were not bent or damaged, and it was no more difficult to drive, regardless of whether you bumped your wheels or even lost a wheel. With increased computing speed, the games can be based on dynamic simulation rather than prearranged scenes. We can expect to see more of these realistic effects in the future: accidents will damage your wheels and your online driving experience will be much more realistic. Realistic modeling and simulation of physics effects are known to demand very large amounts of computing power.

我们在这里提到的所有新应用程序都涉及以不同方式和不同级别模拟物理并发世界，并处理大量数据。事实上，处理海量数据的问题非常普遍，以至于“大数据”一词已成为家喻户晓的词。有了如此大量的数据，大部分计算可以在数据的不同部分上并行完成，尽管它们必须在某些时候进行协调。在大多数情况下，数据交付的有效管理会对并行应用程序的可实现速度产生重大影响。虽然这样做的技术通常为一些每天使用此类应用程序的专家所熟知，但绝大多数应用程序开发人员可以从对这些技术的更直观的理解和实际工作知识中受益。

All the new applications that we mention here involve simulating a physical, concurrent world in different ways and at different levels, with tremendous amounts of data being processed. In fact, the problem of handling massive amounts of data is so prevalent that the term big data has become a household word. And with this huge quantity of data, much of the computation can be done on different parts of the data in parallel, although they will have to be reconciled at some point. In most cases, effective management of data delivery can have a major impact on the achievable speed of a parallel application. While techniques for doing so are often well known to a few experts who work with such applications on a daily basis, the vast majority of application developers can benefit from more intuitive understanding and practical working knowledge of these techniques.

我们的目标是以直观的方式向应用程序开发人员展示数据管理技术，这些开发人员的正规教育可能不是计算机科学或计算机工程。我们还旨在提供许多实用的代码示例和实践练习，帮助读者获得工作知识，这需要一个实用的编程模型来促进并行实现并支持数据交付的正确管理。 CUDA 提供了这样的编程模型，并且已经过大型开发者社区的充分测试。

We aim to present the data management techniques in an intuitive way to application developers whose formal education may not be in computer science or computer engineering. We also aim to provide many practical code examples and hands-on exercises that help readers acquire working knowledge, which requires a practical programming model that facilitates parallel implementation and supports proper management of data delivery. CUDA offers such a programming model and has been well tested by a large developer community.

1.4 加速实际应用

1.4 Speeding Up Real Applications

并行化应用程序可以预期加速多少倍？这取决于应用程序中可以并行化的部分。如果可并行部分所花费的时间百分比为 30%，则并行部分加速 100 倍将减少执行时间不超过 29.7%。整个应用程序的加速仅约为 1.4 倍。事实上，即使并行部分无限加速，也只能减少 30% 的执行时间，实现不超过 1.43 倍的加速。另一方面，如果 99% 的执行时间都在并行部分，则 100 倍的加速会将应用程序执行时间减少到原始时间的 1.99%。这使整个应用程序加速了 50 倍。因此，对于大规模并行处理器来说，应用程序的绝大多数执行都在并行部分进行，以有效加速其执行速度，这一点非常重要。

How many times speedup can be expected from parallelizing an application? It depends on the portion of the application that can be parallelized. If the percentage of time spent in the part that can be parallelized is 30%, a 100× speedup of the parallel portion will reduce the execution time by no more than 29.7%. The speedup for the entire application will be only about 1.4×. In fact, even an infinite amount of speedup in the parallel portion can only slash 30% off execution time, achieving no more than 1.43× speedup. On the other hand, if 99% of the execution time is in the parallel portion, a 100× speedup will reduce the application execution to 1.99% of the original time. This gives the entire application a 50× speedup. Therefore, it is very important that an application has the vast majority of its execution in the parallel portion for a massively parallel processor to effectively speed up its execution.

研究人员已经在某些应用程序中实现了超过 100 倍的加速。然而，这通常只有经过广泛的算法增强后的优化和调优，使得99.9%以上的应用程序执行时间是并行执行。在实践中，应用程序的直接并行化通常会使内存 (DRAM) 带宽饱和，导致仅获得约 10 倍的加速。诀窍在于找出如何解决内存带宽限制，这涉及到进行多种转换之一，以利用专门的 GPU 片上内存来大幅减少对 DRAM 的访问次数。然而，我们必须进一步优化代码，以克服片上内存容量有限等限制。本书的一个重要目标是帮助读者充分理解这些优化并熟练掌握它们。

Researchers have achieved speedups of more than 100× for some applications. However, this is typically achieved only after extensive optimization and tuning after the algorithms have been enhanced, so that more than 99.9% of the application execution time is in parallel execution. In practice, straightforward parallelization of applications often saturates the memory (DRAM) bandwidth, resulting in only about a 10× speedup. The trick is to figure out how to get around memory bandwidth limitations, which involves doing one of many transformations to utilize specialized GPU on-chip memories to drastically reduce the number of accesses to the DRAM. One must, however, further optimize the code to get around limitations such as limited on-chip memory capacity. An important goal of this book is to help readers fully understand these optimizations and become skilled in them.

请记住，相对于单核 CPU 执行所实现的加速水平也可以反映 CPU 对应用程序的适用性：在某些应用程序中，CPU 的性能非常好，这使得使用 GPU 加速性能变得更加困难。大多数应用程序都有一些可以由 CPU 更好地执行的部分。因此，必须给CPU一个公平的执行机会，并确保编写的代码能够让GPU补充CPU的执行，从而正确地利用CPU-GPU组合系统的异构并行计算能力。这正是 CUDA 编程模型所提倡的，我们将在书中进一步解释。

Keep in mind that the level of speedup achieved over single-core CPU execution can also reflect the suitability of the CPU to the application: in some applications, CPUs perform very well, making it harder to speed up performance using a GPU. Most applications have portions that can be much better executed by the CPU. Thus, one must give the CPU a fair chance to perform and make sure that code is written so that GPUs complement CPU execution, thus properly exploiting the heterogeneous parallel computing capabilities of the combined CPU–GPU system. This is precisely what the CUDA programming model promotes, as we will further explain in the book.

图 1.3说明了典型应用的主要部分。实际应用程序的大部分代码往往是连续的。这些连续的部分被描述为桃子的“核”区域：试图将并行计算技术应用于这些部分就像咬进桃核一样——这种感觉不太好！这些部分很难并行化。 CPU 在这些部分往往做得非常好。好消息是，这些部分虽然占用了大部分代码，但往往只占超级应用程序执行时间的一小部分。

Figure 1.3 illustrates the main parts of a typical application. Much of a real application’s code tends to be sequential. These sequential parts are illustrated as the “pit” area of the peach: trying to apply parallel computing techniques to these portions is like biting into the peach pit—not a good feeling! These portions are very hard to parallelize. CPUs tend to do a very good job on these portions. The good news is that these portions, although they can take up a large portion of the code, tend to account for only a small portion of the execution time of super-applications.

然后是我们所说的“桃肉”部分。这些部分很容易并行化，一些早期的图形应用程序也是如此。异构计算系统中的并行编程可以极大地提高这些应用程序的质量。如图1.3所示，早期的 GPGPU 仅覆盖肉类部分的一小部分，这类似于最令人兴奋的应用程序的一小部分。正如我们将看到的，CUDA 编程模型旨在涵盖令人兴奋的应用程序的大部分桃肉部分。事实上，正如我们将在第 20 章中讨论的那样，这些编程模型及其底层硬件仍在快速发展，以实现更大的应用程序部分的高效并行化。

Then come what we call the “peach meat” portions. These portions are easy to parallelize, as are some early graphics applications. Parallel programming in heterogeneous computing systems can drastically improve the quality of these applications. As illustrated in Figure 1.3, early GPGPUs cover only a small portion of the meat section, which is analogous to a small portion of the most exciting applications. As we will see, the CUDA programming models are designed to cover a much larger section of the peach meat portions of exciting applications. In fact, as we will discuss in Chapter 20, these programming models and their underlying hardware are still evolving at a fast pace to enable efficient parallelization of even larger sections of applications.

1.5 并行编程语言和模型

1.5 Parallel Programming Languages and Models

在过去的几十年里，已经提出了许多并行编程语言和模型Mattson2004 ]。最广泛使用的是用于可扩展集群计算的消息传递接口 (MPI) [MPI2009] ，以及用于共享内存多处理器系统的OpenMP [Open2005] 。两者都已成为主要计算机供应商支持的标准化编程接口。 OpenMP 实现由编译器和运行时组成。程序员向 OpenMP 编译器指定有关循环的指令（命令）和编译指示（提示）。通过这些指令和编译指示，OpenMP 编译器可以生成并行代码。运行时系统通过管理并行线程和资源来支持并行代码的执行。 OpenMP 最初是为 CPU 执行而设计的。最近，多个计算机供应商提出并支持一种称为 OpenACC 的变体（参见第 15 章），用于对异构计算系统进行编程。

Many parallel programming languages and models have been proposed in the past several decades Mattson2004]. The ones that are the most widely used are Message Passing Interface (MPI) [MPI2009] for scalable cluster computing, and OpenMP [Open2005] for shared-memory multiprocessor systems. Both have become standardized programming interfaces supported by major computer vendors. An OpenMP implementation consists of a compiler and a runtime. A programmer specifies directives (commands) and pragmas (hints) about a loop to the OpenMP compiler. With these directives and pragmas, OpenMP compilers generate parallel code. The runtime system supports the execution of the parallel code by managing parallel threads and resources. OpenMP was originally designed for CPU execution. More recently, a variation called OpenACC (see Chapter 15) has been proposed and supported by multiple computer vendors for programming heterogeneous computing systems.

OpenACC 的主要优点是它提供编译器自动化和运行时支持，以从程序员那里抽象出许多并行编程细节。这种自动化和抽象可以帮助使应用程序代码在不同供应商生产的系统以及同一供应商的不同代系统之间更加可移植。这就是我们在第 15 章中教授 OpenACC 编程的原因。然而，在 OpenACC 中进行有效的编程仍然需要程序员了解所涉及的所有详细的并行编程概念。由于 CUDA 为程序员提供了对这些并行编程细节的明确控制，因此即使对于那些想要使用 OpenMP 和 OpenACC 作为主要编程接口的人来说，它也是一个极好的学习工具。此外，根据我们的经验，OpenACC 编译器仍在不断发展和改进。许多程序员可能需要使用 CDUA 风格的接口来处理 OpenACC 编译器无法满足的部分。

The major advantage of OpenACC is that it provides compiler automation and runtime support for abstracting away many parallel programming details from programmers. Such automation and abstraction can help make the application code more portable across systems produced by different vendors, as well as different generations of systems from the same vendor. This is why we teach OpenACC programming in Chapter 15. However, effective programming in OpenACC still requires the programmers to understand all the detailed parallel programming concepts involved. Because CUDA gives programmers explicit control of these parallel programming details, it is an excellent learning vehicle even for someone who would like to use OpenMP and OpenACC as their primary programming interface. Furthermore, from our experience, OpenACC compilers are still evolving and improving. Many programmers will likely need to use CDUA-style interfaces for parts where OpenACC compilers fall short.

MPI是集群中的计算节点不共享内存的模型[MPI2009]。所有的数据共享和交互都必须通过显式的消息传递来完成。 MPI 在高性能计算 (HPC) 领域取得了成功。用MPI编写的应用程序已在超过10万个节点的集群计算系统上成功运行。如今，许多 HPC 集群都采用异构 CPU-GPU 节点。虽然 CUDA 是与每个节点的有效接口，但大多数应用程序开发人员需要使用 MPI 在集群级别进行编程。因此，并行很重要HPC 程序员了解如何进行 MPI/CUDA 联合编程，这将在第 19 章中介绍。

MPI is a model where computing nodes in a cluster do not share memory [MPI2009]. All data sharing and interaction must be done through explicit message passing. MPI has been successful in high-performance computing (HPC). Applications written in MPI have run successfully on cluster computing systems with more than 100,000 nodes. Today, many HPC clusters employ heterogeneous CPU–GPU nodes. While CUDA is an effective interface with each node, most application developers need to use MPI to program at the cluster level. Therefore, it is important that a parallel programmer in HPC understands how to do joint MPI/CUDA programming, which is presented in Chapter 19.

然而，由于计算节点之间缺乏共享内存，将应用程序移植到 MPI 所需的工作量可能相当大。程序员需要执行域分解，将输入和输出数据划分到集群节点中。基于域分解，程序员还需要调用消息发送和接收函数来管理节点之间的数据交换。另一方面，CUDA 为 GPU 中的并行执行提供共享内存来解决这一难题。至于CPU和GPU通信，CUDA之前在CPU和GPU之间提供非常有限的共享内存能力。程序员需要以类似于“单边”消息传递的方式管理 CPU 和 GPU 之间的数据传输。异构计算系统（例如 GMAC [GCN2010]和 CUDA 4.0）中的全局地址空间和自动数据传输的新运行时支持现已推出。借助 GMAC，CUDA 或 OpenCL 程序员可以将 C 变量和数据结构声明为在 CPU 和 GPU 之间共享。 GMAC 运行时保持一致性，并根据需要代表程序员自动执行优化的数据传输操作。这种支持显着降低了与计算和 I/O 活动重叠的数据传输所涉及的 CUDA 和 OpenCL 编程复杂性。

The amount of effort needed to port an application into MPI, however, can be quite high due to the lack of shared memory across computing nodes. The programmer needs to perform domain decomposition to partition the input and output data into cluster nodes. Based on the domain decomposition, the programmer also needs to call message sending and receiving functions to manage the data exchange between nodes. CUDA, on the other hand, provides shared memory for parallel execution in the GPU to address this difficulty. As for CPU and GPU communication, CUDA previously provided very limited shared memory capability between the CPU and the GPU. The programmers needed to manage the data transfer between the CPU and the GPU in a manner similar to the “one-sided” message passing. New runtime support for global address space and automated data transfer in heterogeneous computing systems, such as GMAC [GCN2010] and CUDA 4.0, are now available. With GMAC, a CUDA or OpenCL programmer can declare C variables and data structures as shared between CPU and GPU. The GMAC runtime maintains coherence and automatically performs optimized data transfer operations on behalf of the programmer on an as-needed basis. Such support significantly reduces the CUDA and OpenCL programming complexity involved in overlapping data transfer with computation and I/O activities.

2009年，包括Apple、Intel、AMD/ATI和NVIDIA在内的几个主要行业参与者联合开发了一种标准化编程模型，称为开放计算语言（OpenCL）[Khronos2009]。与 CUDA 类似，OpenCL 编程模型定义了语言扩展和运行时 API，以允许程序员管理大规模并行处理器中的并行性和数据交付。与 CUDA 相比，OpenCL 更多地依赖于 API，而更少地依赖于 CUDA 的语言扩展。这使得供应商能够快速调整其现有的编译器和工具来处理 OpenCL 程序。 OpenCL 是一种标准化的编程模型，用 OpenCL 开发的应用程序无需修改即可在所有支持 OpenCL 语言扩展和 API 的处理器上正确运行。然而，人们可能需要修改应用程序才能实现新处理器的高性能。

In 2009, several major industry players, including Apple, Intel, AMD/ATI, and NVIDIA, jointly developed a standardized programming model called Open Compute Language (OpenCL) [Khronos2009]. Similar to CUDA, the OpenCL programming model defines language extensions and runtime APIs to allow programmers to manage parallelism and data delivery in massively parallel processors. In comparison to CUDA, OpenCL relies more on APIs and less on language extensions than CUDA. This allows vendors to quickly adapt their existing compilers and tools to handle OpenCL programs. OpenCL is a standardized programming model in that applications developed in OpenCL can run correctly without modification on all processors that support the OpenCL language extensions and API. However, one will likely need to modify the applications to achieve high performance for a new processor.

熟悉 OpenCL 和 CUDA 的人都知道，OpenCL 和 CUDA 的关键概念和功能之间存在显着的相似性。也就是说，CUDA程序员可以以最小的努力学习OpenCL编程。更重要的是，几乎所有技术使用 CUDA 学到的知识可以轻松应用到 OpenCL 编程中。因此，我们在第 14 章介绍 OpenCL ，并解释如何将本书中的关键概念应用到 OpenCL 编程中。

Those who are familiar with both OpenCL and CUDA know that there is a remarkable similarity between the key concepts and features of OpenCL and those of CUDA. That is, a CUDA programmer can learn OpenCL programming with minimal effort. More importantly, virtually all techniques learned using CUDA can be easily applied to OpenCL programming. Therefore, we introduce OpenCL in Chapter 14 and explain how one can apply the key concepts in this book to OpenCL programming.

1.6 总体目标

1.6 Overarching Goals

我们的主要目标是教读者如何对大规模并行处理器进行编程以实现高性能，并且我们的方法不需要大量的硬件专业知识。有人曾经说过，如果不关心性能，并行编程是很容易的。您实际上可以在一小时内编写一个并行程序。但我们将用很多篇幅来讨论开发高性能并行程序的技术。而且，我们相信，一旦您形成正确的洞察力并以正确的方式去做，事情就会变得容易。特别是，我们将重点关注计算思维技术，使您能够以适合高性能并行计算的方式思考问题。

Our primary goal is to teach the readers how to program massively parallel processors to achieve high performance, and our approach will not require a great deal of hardware expertise. Someone once said that if you don’t care about performance, parallel programming is very easy. You can literally write a parallel program in an hour. But we’re going to dedicate many pages to techniques for developing high-performance parallel programs. And, we believe that it will become easy once you develop the right insight and go about it the right way. In particular, we will focus on computational thinking techniques that will enable you to think about problems in ways that are amenable to high-performance parallel computing.

请注意，硬件架构功能有限制。大多数处理器上的高性能并行编程需要了解硬件工作原理。我们可能需要 10 年或更长时间才能构建工具和机器，以便大多数程序员可以在没有这些知识的情况下工作。即使我们有这样的工具，我们怀疑对硬件有更多了解的程序员将能够比那些不了解硬件的程序员更有效地使用这些工具。然而，我们不会将计算机体系结构作为一个单独的主题来教授。相反，我们将教授基本的计算机体系结构知识，作为高性能并行编程技术讨论的一部分。

Note that hardware architecture features have constraints. High-performance parallel programming on most processors will require some knowledge of how the hardware works. It will probably take 10 or more years before we can build tools and machines so that most programmers can work without this knowledge. Even if we have such tools, we suspect that programmers with more knowledge of the hardware will be able to use the tools in a much more effective way than those who do not. However, we will not be teaching computer architecture as a separate topic. Instead, we will teach the essential computer architecture knowledge as part our discussions on high-performance parallel programming techniques.

我们的第二个目标是教授并行编程的正确功能和可靠性，这是并行计算中的一个微妙问题。过去从事并行系统工作的人都知道，仅实现初始性能是不够的。挑战在于以一种可以调试代码并支持用户的方式来实现它。 CUDA 编程模型鼓励使用简单形式的屏障同步和内存一致性来管理并行性。我们将证明，通过关注数据并行性，人们可以在其应用程序中实现高性能和高可靠性。

Our second goal is to teach parallel programming for correct functionality and reliability, which constitute a subtle issue in parallel computing. Those who have worked on parallel systems in the past know that achieving initial performance is not enough. The challenge is to achieve it in such a way that you can debug the code and support users. The CUDA programming model encourages the use of a simple form of barrier synchronization and memory consistency for managing parallelism. We will show that by focusing on data parallelism, one can achieve both high performance and high reliability in their applications.

我们的第三个目标是通过探索并行编程方法来实现未来硬件各代的可扩展性，以便未来的机器（将越来越并行）可以比今天的机器更快地运行代码。我们希望帮助您掌握并行编程，所以您的程序可以扩展到新一代机器的性能水平。这种可扩展性的关键是规范和本地化内存数据访问，以最大限度地减少关键资源的消耗以及访问和更新数据结构的冲突。

Our third goal is scalability across future hardware generations by exploring approaches to parallel programming such that future machines, which will be more and more parallel, can run your code faster than today’s machines. We want to help you master parallel programming so that your programs can scale up to the level of performance of new generations of machines. The key to such scalability is to regularize and localize memory data accesses to minimize consumption of critical resources and conflicts in accessing and updating data structures.

实现这些目标需要大量的技术知识，因此我们将在本书中介绍相当多的并行编程原理和模式。然而，我们不能保证涵盖所有这些内容，因此我们选择了最有用且经过充分验证的技术来详细介绍。为了补充您的知识和专业知识，我们提供了推荐文献列表。现在我们准备向您提供本书其余部分的快速概述。

Much technical knowledge will be required to achieve these goals, so we will cover quite a few principles and patterns of parallel programming in this book. We cannot guarantee that we will cover all of them, however, so we have selected the most useful and well-proven techniques to cover in detail. To complement your knowledge and expertise, we include a list of recommended literature. We are now ready to give you a quick overview of the rest of the book.

1.7 本书的组织

1.7 Organization of the Book

第 2 章回顾了 GPU 计算的历史。首先简要总结了图形硬件向更多可编程性的发展，然后讨论了历史上的 GPGPU 运动。 CUDA 编程模型当前的许多功能和限制都源于这些历史性的发展。充分了解这些历史性发展将有助于读者更好地了解硬件发展的现状和未来趋势，这些发展将继续影响将从 CUDA 中受益的应用程序类型。

Chapter 2 reviews the history of GPU computing. It starts with a brief summary of the evolution of graphics hardware toward more programmability and then discusses the historical GPGPU movement. Many of the current features and limitations of the CUDA programming model find their root in these historic developments. A good understanding of these historic developments will help readers better understand the current state and the future trends of hardware evolution that will continue to impact the types of applications that will benefit from CUDA.

第3章介绍数据并行性和CUDA C编程。本章基于学生之前有过 C 编程经验的事实。它首先介绍了 CUDA C 作为 C 的简单、小型扩展，支持异构 CPU-GPU 联合计算和广泛使用的 SPMD（单程序、多数据）并行编程模型。然后，它涵盖了以下方面的思维过程：(1) 确定应用程序中要并行化的部分； (2)隔离并行化代码要使用的数据，利用API(应用程序编程接口)函数在并行计算设备上分配内存； (3)使用API​​函数向并行计算设备传输数据； (4)开发内核函数，由并行部分的线程执行； (5)启动内核函数以供并行线程执行； (6) 最终通过 API 函数调用将数据传输回主机处理器。

Chapter 3 introduces data parallelism and the CUDA C programming. This chapter relies on the fact that students have had previous experience with C programming. It first introduces CUDA C as a simple, small extension to C that supports heterogeneous CPU–GPU joint computing and the widely used SPMD (single program, multiple data) parallel programming model. It then covers the thought process involved in (1) identifying the part of application programs to be parallelized; (2) isolating the data to be used by the parallelized code, using an API (Application Programming Interface) function to allocate memory on the parallel computing device; (3) using an API function to transfer data to the parallel computing device; (4) developing a kernel function that will be executed by threads in the parallelized part; (5) launching a kernel function for execution by parallel threads; and (6) eventually transferring the data back to the host processor with an API function call.

虽然第 3 章的目标是教授 CUDA C 编程模型的足够概念，以便学生能够编写简单的并行 CUDA C 程序，但它实际上涵盖了以下几个基本技能：基于任何并行编程模型开发并行应用程序。我们使用向量加法的运行示例来使本章具体化。我们还将 CUDA 与其他并行编程模型（包括 OpenMP 和 OpenCL）进行了比较。

While the objective of Chapter 3 is to teach enough concepts of the CUDA C programming model so that the students can write a simple parallel CUDA C program, it actually covers several basic skills needed to develop a parallel application based on any parallel programming model. We use a running example of vector addition to make this chapter concrete. We also compare CUDA with other parallel programming models including OpenMP and OpenCL.

第 4 章介绍了 CUDA 并行执行模型的更多细节。它对线程的创建、组织、资源绑定、数据绑定和调度提供了足够的洞察力，使读者能够使用 CUDA C 实现复杂的计算，并推理其 CUDA 代码的性能行为。第 5 章专门介绍可用于保存 CUDA 变量以管理数据传输和提高程序执行速度的特殊存储器。

Chapter 4 presents more details of the parallel execution model of CUDA. It gives enough insight into the creation, organization, resource binding, data binding, and scheduling of threads to enable readers to implement sophisticated computation using CUDA C and reason about the performance behavior of their CUDA code. Chapter 5 is dedicated to the special memories that can be used to hold CUDA variables for managing data delivery and improving program execution speed.

第 6 章介绍了当前 CUDA 硬件中的几个重要的性能考虑因素。特别是，它提供了有关线程执行、内存数据访问和资源分配的更多详细信息。这些细节构成了程序员推理其组织计算和数据决策结果的概念基础。

Chapter 6 presents several important performance considerations in current CUDA hardware. In particular, it gives more details in thread execution, memory data accesses, and resource allocation. These details form the conceptual basis for programmers to reason about the consequence of their decisions on organizing their computation and data.

第 7 章介绍浮点数格式、精度和准确度的概念。它说明了为什么不同的并行执行安排会导致不同的输出值。它还教授数值稳定性的概念以及在并行算法中保持数值稳定性的实用技术。

Chapter 7 introduces the concepts of floating-point number format, precision, and accuracy. It shows why different parallel execution arrangements can result in different output values. It also teaches the concept of numerical stability and practical techniques for maintaining numerical stability in parallel algorithms.

第 8 - 10章介绍了三种重要的并行计算模式，使读者能够更深入地了解并行编程技术和并行执行机制。第 8 章介绍了卷积，这是一种常用的并行计算模式，需要仔细管理数据访问局部性。我们还使用这种模式在现代 GPU 中引入恒定内存和缓存。第9章介绍了前缀和或扫描，这是一种重要的并行计算模式，它将顺序计算转换为并行计算。我们还使用这种模式来引入并行算法中工作效率的概念。第 10 章介绍了稀疏矩阵计算，这是一种用于处理非常大的数据集的模式。本章向读者介绍重新排列数据以实现更高效的并行访问的概念：填充、排序、转置和正则化。

Chapter 8-10 present three important parallel computation patterns that give readers more insight into parallel programming techniques and parallel execution mechanisms. Chapter 8 presents convolution, a frequently used parallel computing pattern that requires careful management of data access locality. We also use this pattern to introduce constant memory and caching in modern GPUs. Chapter 9 presents prefix sum, or scan, an important parallel computing pattern that coverts sequential computation into parallel computation. We also use this pattern to introduce the concept of work efficiency in parallel algorithms. Chapter 10 presents sparse matrix computation, a pattern used for processing very large data sets. This chapter introduces readers to the concepts of rearranging data for more efficient parallel access: padding, sorting, transposition, and regularization.

虽然这些章节基于 CUDA，但它们可以帮助读者建立一般并行编程的基础。我们相信，当我们自下而上学习时，人类理解得最好。也就是说，我们必须首先在特定编程模型的上下文中学习概念，这为我们将知识推广到其他编程模型时提供了坚实的基础。当我们这样做时，我们可以从 CUDA 模型中汲取具体经验。深入体验与CUDA模型的接触也使我们变得成熟，这将帮助我们学习甚至可能与CUDA模型无关的概念。

While these chapters are based on CUDA, they help readers build up the foundation for parallel programming in general. We believe that humans understand best when we learn from the bottom up. That is, we must first learn the concepts in the context of a particular programming model, which provides us with solid footing when we generalize our knowledge to other programming models. As we do so, we can draw on our concrete experience from the CUDA model. An in-depth experience with the CUDA model also enables us to gain maturity, which will help us learn concepts that may not even be pertinent to the CUDA model.

第 11 章和第12 章是两个实际应用程序的案例研究，引导读者了解并行化和优化其应用程序以显着加速的思维过程。对于每个应用程序，我们首先确定制定并行执行基本结构的替代方法，然后推理每个替代方案的优点和缺点。然后，我们完成实现高性能所需的代码转换步骤。这两章帮助读者将前面章节中的所有材料放在一起，为自己的应用程序开发项目做好准备。

Chapters 11 and 12 are case studies of two real applications, which take readers through the thought process of parallelizing and optimizing their applications for significant speedups. For each application, we start by identifying alternative ways of formulating the basic structure of the parallel execution and follow up with reasoning about the advantages and disadvantages of each alternative. We then go through the steps of code transformation needed to achieve high performance. These two chapters help readers put all the materials from the previous chapters together and prepare for their own application development projects.

第13章将并行编程技术概括为问题分解原理、算法策略和计算思维。它通过涵盖组织程序的计算任务以使它们可以并行完成的概念来实现这一点。我们首先讨论将抽象科学概念组织成计算任务的转化过程，这是生产高质量应用软件（串行或并行）的重要的第一步。然后讨论并行算法结构及其对应用程序性能的影响，这是基于 CUDA 的性能调优经验。本章最后讨论了并行编程风格和模型，使读者能够将他们的知识置于更广泛的背景中。通过本章，读者可以开始从 SPMD 编程风格推广到其他并行编程风格，例如 OpenMP 中的循环并行和 p 线程编程中的 fork-join。尽管我们不会讨论这些替代的并行编程风格，但我们希望读者能够利用在本书中获得的基础来学习其中任何一种编程风格。

Chapter 13 generalizes the parallel programming techniques into problem decomposition principles, algorithm strategies, and computational thinking. It does so by covering the concept of organizing the computation tasks of a program so that they can be done in parallel. We start by discussing the translational process of organizing abstract scientific concepts into computational tasks, which is an important first step in producing quality application software, serial or parallel. It then discusses parallel algorithm structures and their effects on application performance, which is grounded in the performance tuning experience with CUDA. The chapter concludes with a treatment of parallel programming styles and models, enabling readers to place their knowledge in a wider context. With this chapter, readers can begin to generalize from the SPMD programming style to other styles of parallel programming, such as loop parallelism in OpenMP and fork-join in p-thread programming. Although we do not go into these alternative parallel programming styles, we expect that readers will be able to learn to program in any of them with the foundation they gain in this book.

第14章从CUDA程序员的角度介绍了OpenCL编程模型。读者会发现 OpenCL 与 CUDA 极其相似。最重要的区别在于 OpenCL 使用 API 函数来实现内核启动和线程识别等功能。 API函数的使用使得OpenCL使用起来更加繁琐。尽管如此，CUDA 程序员拥有理解和编写 OpenCL 程序所需的所有知识和技能。事实上，我们认为教授 OpenCL 编程的最佳方法是先教授 CUDA。我们用一章来演示这一点，该章将所有主要 OpenCL 功能与其相应的 CUDA 功能联系起来。我们还通过将简单的 CUDA 示例改编为 OpenCL 来说明这些功能的使用。

Chapter 14 introduces the OpenCL programming model from a CUDA programmer’s perspective. Readers will find OpenCL to be extremely similar to CUDA. The most important difference arises from OpenCL’s use of API functions to implement functionalities such as kernel launching and thread identification. The use of API functions makes OpenCL more tedious to use. Nevertheless, a CUDA programmer has all the knowledge and skills needed to understand and write OpenCL programs. In fact, we believe that the best way to teach OpenCL programming is to teach CUDA first. We demonstrate this with a chapter that relates all major OpenCL features to their corresponding CUDA features. We also illustrate the use of these features by adapting our simple CUDA examples into OpenCL.

第 15 章介绍了 OpenACC 编程接口。它展示了如何使用指令和编译指示告诉编译器可以并行化循环，并在需要时指示编译器如何并行化循环。它还使用具体示例来说明如何利用该接口并使代码在供应商系统之间更加可移植。通过本书中的基本概念，读者会发现 OpenACC 编程指令和编译指示很容易学习和掌握。

Chapter 15 presents the OpenACC programming interface. It shows how to use directives and pragmas to tell the compiler that a loop can be parallelized, and if desirable, instruct the compiler how to parallelize the loop. It also uses concrete examples to illustrate how one can take advantage of the interface and make their code more portable across vendor systems. With the foundational concepts in this book, readers will find the OpenACC programming directives and pragmas easy to learn and master.

第 16 章介绍了 Thrust，这是一个面向生产力的 C++ 库，用于构建 CUDA 应用程序。本章展示了如何使用现代面向对象编程接口和技术来提高并行编程环境中的生产力。特别是，它展示了通用编程和抽象如何显着减少应用程序的工作量和代码复杂性。

Chapter 16 covers Thrust, a productivity-oriented C++ library for building CUDA applications. This is a chapter that shows how modern object-oriented programming interfaces and techniques can be used to increase productivity in a parallel programming environment. In particular, it shows how generic programming and abstractions can significantly reduce the efforts and code complexity of applications.

第17章介绍了CUDA FORTRAN，这是一个基于CUDA模型支持FORTRAN风格编程的接口。使用 CUDA C 学习的所有概念和技术都可以在 CUDA 中编程时应用。此外，CUDA FORTRAN 接口对多维数组有强大的支持，使 3D 模型的编程更具可读性。它还采用 FORTRAN 数组数据布局约定，并且可以更好地与用 FORTRAN 编写的现有应用程序配合使用。

Chapter 17 presents CUDA FORTRAN, an interface that supports FORTRAN-style programming based on the CUDA model. All concepts and techniques learned using CUDA C can be applied when programming in CUDA. In addition, the CUDA FORTRAN interface has strong support for multidimensional arrays that make programming of 3D models much more readable. It also assumes the FORTRAN array data layout convention and works better with an existing application written in FORTRAN.

第 18 章概述了 Microsoft 的 C++AMP 编程接口。该编程接口使用组合语言扩展和 API 支持来支持数据并行计算模式。它允许程序员使用 C++ 功能来提高生产力。与 OpenACC 一样，C++AMP 抽象出了一些特定于硬件的并行编程细节，因此代码可能更容易跨供应商系统移植。

Chapter 18 is an overview of the C++AMP programming interface from Microsoft. This programming interface uses a combination language extension and API support to support data-parallel computation patterns. It allows programmers to use C++ features to increase their productivity. Like OpenACC, C++AMP abstracts away some of the parallel programming details that are specific to the hardware so the code is potentially more portable across vendor systems.

第 19 章介绍了 MPI/CUDA 联合编程。我们介绍了程序员需要了解的关键 MPI 概念，以将其异构应用程序扩展到集群环境中的多个节点。特别是，我们将重点关注将 CUDA 内核扩展到多个节点的背景下的域分区、点对点通信和集体通信。

Chapter 19 presents an introduction to joint MPI/CUDA programming. We cover the key MPI concepts that a programmer needs to understand to scale their heterogeneous applications to multiple nodes in a cluster environment. In particular, we will focus on domain partitioning, point-to-point communication, collective communication in the context of scaling a CUDA kernel into multiple nodes.

第 20 章介绍了 Kepler GPU 及其后续产品中可用的动态并行功能。动态并行性可能有助于复杂算法的实现，以减少 CPU-GPU 交互开销，释放 CPU 用于其他任务，并提高 GPU 执行资源的利用率。我们描述了动态并行性的基本概念以及为什么某些算法可以从动态并行性中受益。然后我们用一个例子来说明动态并行性的用法小型设计的代码示例以及更复杂的实际代码示例。

Chapter 20 introduces the dynamic parallelism capability available in the Kepler GPUs and their successors. Dynamic parallelism can potentially help the implementations of sophisticated algorithms to reduce CPU-GPU interaction overhead, free up CPU for other tasks, and improve the utilization of GPU execution resources. We describe the basic concepts of dynamic parallelism and why some algorithms can benefit from dynamic parallelism. We then illustrate the usage of dynamic parallelism with a small contrived code example as well as a more complex realistic code example.

第 21 章提供了一些结论以及对大规模并行编程的未来的展望。我们首先重新审视我们的目标，并总结各章如何组合在一起以帮助实现目标。然后，我们简要调查了大规模并行处理器架构的主要趋势，以及这些趋势将如何影响未来的并行编程。最后，我们预测大规模并行计算的快速进步将使其成为未来十年最令人兴奋的领域之一。

Chapter 21 offers some concluding remarks and an outlook for the future of massively parallel programming. We first revisit our goals and summarize how the chapters fit together to help achieve the goals. We then present a brief survey of the major trends in the architecture of massively parallel processors and how these trends will likely impact parallel programming in the future. We conclude with a prediction that these fast advances in massively parallel computing will make it one of the most exciting areas in the coming decade.

参考

References

1. Gelado, I.、Cabezas, J.、Navarro, N.、Stone, JE、Patel, SJ 和 Hwu, WW 异构并行系统的异步分布式共享内存模型，编程语言和操作的架构支持国际会议Systems，2010 年 3 月。技术报告，IMPACT Group，伊利诺伊大学厄巴纳-香槟分校。

1. Gelado, I., Cabezas, J., Navarro, N., Stone, J. E., Patel, S. J., & Hwu, W. W. An Asynchronous Distributed Shared Memory Model for Heterogeneous Parallel Systems, International Conference on Architectural Support for Programming Languages and Operating Systems, March 2010. Technical Report, IMPACT Group, University of Illinois, Urbana-Champaign.

2. Hwu WW、Keutzer K、Mattson T。并发挑战。IEEE 计算机设计和测试， 2008 年 7 月/8 月：312–320。

2. Hwu WW, Keutzer K, Mattson T. The Concurrency Challenge. IEEE Design and Test of Computers, July/August 2008:312–320.

3. Khronos Group，OpenCL 规范版本 1.0，可从以下网址获取：( http://www.khronos.org/registry/cl/specs/opencl-1.0.29.pdf )。

3. The Khronos Group, The OpenCL Specification Version 1.0, Available at: (http://www.khronos.org/registry/cl/specs/opencl-1.0.29.pdf).

4. 马特森 TG、桑德斯 BA、马辛吉尔 BL。并行编程模式波士顿：Addison-Wesley Professional； 2004年。

4. Mattson TG, Sanders BA, Massingill BL. Patterns of Parallel Programming Boston: Addison-Wesley Professional; 2004.

5. 消息传递接口论坛，“MPI — 消息传递接口标准版本 2.2”，网址为：< http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf >。，9 月2009 年 4 月。

5. Message Passing Interface Forum, “MPI—A Message Passing Interface Standard Version 2.2,” Available at: <http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf>., Sept. 4, 2009.

6. NVIDIA 公司，NVIDIA CUDA 计算统一设备架构编程指南 1.0，2007 年 6 月，可在以下网址获取：< http://www.cs.berkeley.edu/~yelick/cs194f07/handouts/NVIDIA_CUDA_Programming_Guide.pdf >。

6. NVIDIA Corporation, NVIDIA CUDA Compute Unified Device Architecture Programming Guide 1.0, June 2007, Available at: <http://www.cs.berkeley.edu/~yelick/cs194f07/handouts/NVIDIA_CUDA_Programming_Guide.pdf>.

7. OpenMP 架构审查委员会，“OpenMP 应用程序接口版本 3.1”。 2011 年 7 月，可访问：< http://www.openmp.org/mp-documents/OpenMP3.1.pdf >。

7. OpenMP Architecture Review Board, “OpenMP Application Program Interface Version 3.1.” July 2011, Available at: <http://www.openmp.org/mp-documents/OpenMP3.1.pdf>.

8. Sutter H、Larus J。软件和并发革命。ACM 队列。 2005 年 9 月；3(7)：54–62。

8. Sutter H, Larus J. Software and the Concurrency Revolution. ACM Queue. Sept. 2005;3(7):54–62.

9. von Neumann J. EDVAC 报告初稿。见：Goldstine HH，编辑。计算机：从帕斯卡到冯·诺依曼。新泽西州普林斯顿：普林斯顿大学出版社； 1972年。

9. von Neumann J. First Draft of a Report on the EDVAC. In: Goldstine HH, ed. The Computer: From Pascal to von Neumann. Princeton, NJ: Princeton University Press; 1972.

10. Wing J. 计算思维。ACM 的通讯。 2006；49(3)：33-35。

10. Wing J. Computational Thinking. Communications of the ACM. 2006;49(3):33–35.

2.1 图形管线的演变

2.1 Evolution of Graphics Pipelines

三维 (3D) 图形管道硬件从 20 世纪 80 年代初的大型昂贵系统发展到小型工作站，再到 20 世纪 90 年代中后期的 PC 加速器。在此期间，性能领先的图形子系统的价格从 50,000 美元下降到 500 美元。同期，性能从每秒 5000 万像素提升到每秒 10 亿像素，从每秒 10 万个顶点提升到每秒 1000 万个顶点。虽然这些这些进步与半导体器件特征尺寸的不断缩小有很大关系，也来自图形算法和硬件设计创新的新创新。这些创新塑造了现代 GPU 的原生硬件功能。

Three-dimensional (3D) graphics pipeline hardware evolved from the large expensive systems of the early 1980s to small workstations and then PC accelerators in the mid- to late 1990s. During this period, the performance-leading graphics subsystems declined in price from $50,000 to $500. During the same period, the performance increased from 50 million pixels per second to 1 billion pixels per second, and from 100,000 vertices per second to 10 million vertices per second. While these advancements have much to do with the relentlessly shrinking feature sizes of semiconductor devices, they also come from the new innovations of graphics algorithms and hardware design innovations. These innovations have shaped the native hardware capabilities of modern GPUs.

计算机应用中对高质量实时图形的市场需求推动了图形硬件性能的显着进步。例如，在电子游戏应用中，需要以每秒 60 帧的速率以不断增加的分辨率渲染越来越复杂的场景。最终结果是，在过去 30 年中，图形架构已经从用于绘制线框图的简单管道发展为由多个能够渲染 3D 场景的复杂交互式图像的深度并行管道组成的高度并行设计。同时，涉及的许多硬件功能变得更加复杂且用户可编程。

The remarkable advancement of graphics hardware performance has been driven by the market demand for high-quality real-time graphics in computer applications. For example, in an electronic gaming application, one needs to render evermore complex scenes at ever-increasing resolution at a rate of 60 frames per second. The net result is that over the last 30 years, graphics architecture has evolved from a simple pipeline for drawing wireframe diagrams to a highly parallel design consisting of several deep parallel pipelines capable of rendering complex interactive imagery of 3D scenes. Concurrently, many of the hardware functionalities involved became far more sophisticated and user programmable.

2.2 GPGPU：中间步骤

2.2 GPGPU: An Intermediate Step

虽然 GPU 硬件设计朝着更加统一的处理器发展，但它越来越类似于高性能并行计算机。随着支持 DirectX 9 的 GPU 的出现，一些研究人员注意到 GPU 的原始性能增长路径，并开始探索使用 GPU 来解决计算密集型科学和工程问题。然而，DirectX 9 GPU 的设计只是为了满足图形 API 所需的功能。为了访问计算资源，程序员必须将他或她的问题转化为图形操作，以便可以通过 OpenGL 或 DirectX API 调用启动计算。例如，要同时运行计算函数的多个实例，必须将其编写为像素着色器。输入数据的集合必须存储在纹理图像中，并通过提交三角形（如果需要的话，可以裁剪为矩形形状）将其发送到 GPU。输出必须转换为光栅操作生成的一组像素。

While the GPU hardware design evolved toward more unified processors, it increasingly resembled high-performance parallel computers. As DirectX 9–capable GPUs became available, some researchers took notice of the raw performance growth path of GPUs and they started to explore the use of GPUs to solve compute-intensive science and engineering problems. However, DirectX 9 GPUs had been designed only to match the features required by the graphics APIs. To access the computational resources, a programmer had to cast his or her problem into graphics operations so that the computation could be launched through OpenGL or DirectX API calls. For example, to run many simultaneous instances of a compute function, it had to be written as a pixel shader. The collection of input data had to be stored in texture images and issued to the GPU by submitting triangles (with clipping to a rectangle shape if that’s what was desired). The output had to be cast as a set of pixels generated from the raster operations.

事实上，GPU 处理器阵列和帧缓冲存储器接口被设计用于处理图形数据，这对于一般的数值应用程序来说限制太大。具体地，着色器程序的输出数据是其存储位置已被预先确定的单个像素。因此，图形处理器阵列的设计受到很大限制内存读写能力。图2.6说明了早期可编程着色器处理器阵列的有限存储器访问能力；着色器程序员需要使用纹理来访问其输入数据的任意内存位置。更重要的是，着色器无法使用计算出的内存地址执行写入操作（称为分散操作）。将结果写入内存的唯一方法是将其作为像素颜色值发出，并配置帧缓冲区操作阶段以将结果写入（或混合，如果需要）到 2D 帧缓冲区。

The fact that the GPU processor array and frame buffer memory interface were designed to process graphics data proved too restrictive for general numeric applications. In particular, the output data of the shader programs are single pixels of which the memory location has been predetermined. Thus, the graphics processor array is designed with very restricted memory reading and writing capability. Figure 2.6 illustrates the limited memory access capability of early programmable shader processor arrays; shader programmers needed to use texture to access arbitrary memory locations for their input data. More importantly, shaders did not have the means to perform writes with calculated memory addresses, referred to as scatter operations, to memory. The only way to write a result to memory was to emit it as a pixel color value, and configure the frame buffer operation stage to write (or blend, if desired) the result to a 2D frame buffer.

此外，从一次计算到下一次计算获得结果的唯一方法是将所有并行结果写入像素帧缓冲区，然后使用该帧缓冲区作为纹理贴图作为下一阶段像素片段着色器的输入。计算。也不支持一般用户定义的数据类型——大多数数据必须存储在一分量、二分量或四分量向量数组中。在这个时代，将通用计算映射到 GPU 是相当尴尬的。尽管如此，勇敢的研究人员通过艰苦的努力展示了一些有用的应用程序。这个领域被称为 GPGPU，即 GPU 上的通用计算。

Furthermore, the only way to get a result from one pass of computation to the next was to write all parallel results to a pixel frame buffer, then use that frame buffer as a texture map as input to the pixel fragment shader of the next stage of the computation. There was also no support for general user-defined data types—most data had to be stored in one-, two-, or four-component vector arrays. Mapping general computations to a GPU in this era was quite awkward. Nevertheless, intrepid researchers demonstrated a handful of useful applications with painstaking efforts. This field was called GPGPU, for general-purpose computing on GPUs.

2.3 GPU计算

2.3 GPU Computing

在开发 Tesla GPU 架构时，NVIDIA 意识到，如果程序员能够想到，它的潜在用处将会大得多。GPU就像处理器一样。 NVIDIA 选择了一种编程方法，程序员将明确声明其工作负载的数据并行方面。

While developing the Tesla GPU architecture, NVIDIA realized its potential usefulness would be much greater if programmers could think of the GPU like a processor. NVIDIA selected a programming approach in which programmers would explicitly declare the data-parallel aspects of their workload.

对于 DirectX 10 代显卡，NVIDIA 已经开始开发高效浮点和整数处理器，该处理器可以同时运行各种工作负载以支持逻辑图形管道。 Tesla 架构 GPU 的设计者又迈出了一步。着色器处理器成为具有指令存储器、指令高速缓存和指令排序控制逻辑的完全可编程处理器。通过让多个着色器处理器共享其指令缓存和指令排序控制逻辑，可以降低这些额外硬件资源的成本。这种设计风格非常适合图形应用程序，因为需要将相同的着色器程序应用于大量顶点或像素。 NVIDIA 添加了具有随机字节寻址功能的内存加载和存储指令，以支持编译的 C 程序的要求。对于非图形应用程序程序员来说，Tesla 架构 GPU 引入了一种更通用的并行编程模型，具有并行线程层次结构、屏障同步和原子操作来调度和管理高度并行计算工作。 NVIDIA 还开发了 CUDA C/C++ 编译器、库和运行时软件，使程序员能够轻松访问新的数据并行计算模型并开发应用程序。程序员不再需要使用图形API来访问GPU并行计算能力。 G80芯片基于Tesla架构，用于NVIDIA的GeForce 8800 GTX。 G80 之后是 G92、GT200、费米和开普勒。

For the DirectX 10–generation graphics, NVIDIA had already begun work on a high-efficiency floating-point and integer processor that could run a variety of simultaneous workloads to support the logical graphics pipeline. The designers of the Tesla architecture GPUs took another step. The shader processors became fully programmable processors with instruction memory, instruction cache, and instruction sequencing control logic. The cost of these additional hardware resources was reduced by having multiple shader processors to share their instruction cache and instruction sequencing control logic. This design style works well with graphics applications because the same shader program needs to be applied to a massive number of vertices or pixels. NVIDIA added memory load and store instructions with random byte addressing capability to support the requirements of compiled C programs. To nongraphics application programmers, the Tesla architecture GPUs introduced a more generic parallel programming model with a hierarchy of parallel threads, barrier synchronization, and atomic operations to dispatch and manage highly parallel computing work. NVIDIA also developed the CUDA C/C++ compiler, libraries, and runtime software to enable programmers to readily access the new data-parallel computation model and develop applications. Programmers no longer need to use the graphics API to access the GPU parallel computing capabilities. The G80 chip was based on the Tesla architecture and was used in NVIDIA’s GeForce 8800 GTX. G80 was followed later by G92, GT200, Fermi, and Kepler.

参考文献和进一步阅读

References and Further Reading

1. Akeley K、Jermoluk T。高性能多边形渲染。程序 SIGGRAPH 1988 1998：239–246。

1. Akeley K, Jermoluk T. High-Performance polygon rendering. Proc SIGGRAPH 1988 1998:239–246.

2. Akeley K. RealityEngine 图形。程序 SIGGRAPH 1993 1993：109–116。

2. Akeley K. RealityEngine graphics. Proc SIGGRAPH 1993 1993:109–116.

3. 布莱洛赫英国。前缀和及其应用。见：Reif JH，编辑。并行算法的综合。旧金山：摩根·考夫曼； 1990年。

3. Blelloch GB. Prefix sums and their applications. In: Reif JH, ed. Synthesis of parallel algorithms. San Francisco: Morgan Kaufmann; 1990.

4. Blythe D. direct3D 10 系统。ACM 跨图形。 2006；25(3)：724–734。

4. Blythe D. The direct3D 10 system. ACM Trans Graphics. 2006;25(3):724–734.

5. Buck I、Foley T、Horn D 等。 Brook for GPU：图形硬件上的流计算。Proc SIGGRAPH 2004 2004:777–786 也可从以下网址获取： < http://doi.acm.org/10.1145/1186562.1015800 >。

5. Buck I, Foley T, Horn D, et al. Brook for GPUs: Stream computing on graphics hardware. Proc SIGGRAPH 2004 2004:777–786 also Available at: <http://doi.acm.org/10.1145/1186562.1015800>.

6. G 长老“Radeon 9700”，eurographics/SIGGRAPH 图形硬件研讨会。Hot3D 会议2002；网址：< http://www.graphicshardware.org/previous/www_2002/presentations/Hot3D-RADEON9700.ppt >。

6. Elder G. “Radeon 9700,” eurographics/SIGGRAPH workshop on graphics hardware. Hot3D Session 2002; Available at: <http://www.graphicshardware.org/previous/www_2002/presentations/Hot3D-RADEON9700.ppt>.

7.费尔南多·R，基尔加德·MJ。 Cg 教程：可编程实时图形的权威指南。马萨诸塞州雷丁：艾迪生-韦斯利； 2003年。

7. Fernando R, Kilgard MJ. The Cg tutorial: The definitive guide to programmable real-time graphics. Reading, MA: Addison-Wesley; 2003.

8.费尔南多·R，编辑。Gpu gems：实时图形的编程技术、提示和技巧。马萨诸塞州雷丁：艾迪生-韦斯利； 2004年；也可访问：< http://developer.nvidia.com/object/gpu_gems_home.html >。

8. Fernando R, ed. Gpu gems: Programming techniques, tips, and tricks for real-time graphics. Reading, MA: Addison-Wesley; 2004; also Available at: <http://developer.nvidia.com/object/gpu_gems_home.html>.

9. Foley J、van Dam A、Feiner S、Hughes J。计算机图形学：原理与实践，C 阅读第二版，马萨诸塞州：Addison-Wesley； 1995年。

9. Foley J, van Dam A, Feiner S, Hughes J. Computer graphics: Principles and practice, second edition in C Reading, MA: Addison-Wesley; 1995.

10.希利斯·WD，斯蒂尔·GL。数据并行算法。社区 ACM。 1986;29(12):1170–1183< http://doi.acm.org/10.1145/7902.7903 >。

10. Hillis WD, Steele GL. Data parallel algorithms. Commun ACM. 1986;29(12):1170–1183 <http://doi.acm.org/10.1145/7902.7903>.

11. IEEE 754R 工作组。浮点运算 P754 的草案标准。 < http://www.validlab.com/754R/drafts/archive/2006-10-04.pdf >。

11. IEEE 754R working group. DRAFT standard for floating-point arithmetic P754. <http://www.validlab.com/754R/drafts/archive/2006-10-04.pdf>.

12.工业光与魔法。OpenEXR 2003；网址：< //www.openexr.com >。

12. Industrial light and magic. OpenEXR 2003; Available at: <//www.openexr.com>.

13.英特尔公司。Intel 64 和 IA-32 架构优化参考手册2007；网址：< http://www3.intel.com/design/processor/manuals/248966.pdf >。

13. Intel Corporation. Intel 64 and IA-32 Architectures Optimization Reference Manual 2007; Available at: <http://www3.intel.com/design/processor/manuals/248966.pdf>.

14. Kessenich J. Opengl 着色语言，语言版本 1.20 2006；位于：< http://www.opengl.org/documentation/specs/ >。

14. Kessenich J. The Opengl Shading Language, Language Version 1.20 2006; Available at: <http://www.opengl.org/documentation/specs/>.

15. Kirk D、Voorhies D。DN10000VS 的渲染架构。SIGGRAPH 程序 1990 1990；299–307。

15. Kirk D, Voorhies D. The rendering architecture of the DN10000VS. Proc SIGGRAPH 1990 1990;299–307.

16. Lindholm E、Kilgard MJ、Moreton H。用户可编程顶点引擎。程序 SIGGRAPH 2001 2001；149–158。

16. Lindholm E, Kilgard MJ, Moreton H. A user-programmable vertex engine. Proc SIGGRAPH 2001 2001;149–158.

17. Lindholm E、Nickolls J、Oberman S、Montrym J。NVIDIA 特斯拉：统一的图形和计算架构。IEEE 微. 2008；28(2):39–55。

17. Lindholm E, Nickolls J, Oberman S, Montrym J. NVIDIA tesla: A unified graphics and computing architecture. IEEE Micro. 2008;28(2):39–55.

18. 微软公司。 Microsoft DirectX 规范，可从以下位置获取： < http://msdn.microsoft.com/directx/ > 。

18. Microsoft corporation. Microsoft DirectX Specification, Available at: <http://msdn.microsoft.com/directx/>.

19. 微软公司。 Microsoft directx 9 可编程图形管道 华盛顿州雷德蒙德：微软出版社； 2003年。

19. Microsoft Corporation. Microsoft directx 9 programmable graphics pipeline Readmond, WA: Microsoft Press; 2003.

20. Montrym J、Baum D、Dignam D、Migdal C。InfiniteReality：实时图形系统。SIGGRAPH 程序 1997 1997；293–301。

20. Montrym J, Baum D, Dignam D, Migdal C. InfiniteReality: A real-time graphics system. Proc SIGGRAPH 1997 1997;293–301.

21. Montrym J、Moreton H。GeForce 6800。IEEE Micro。 2005；25(2)：41-51。

21. Montrym J, Moreton H. The GeForce 6800. IEEE Micro. 2005;25(2):41–51.

22.摩尔·GE。将更多元件塞进集成电路中。电子产品。 1965;38 可在 < http://download.intel.com/museum/Moores_Law/Articles-Press_Releases/Gordon_Moore_1965_Article.pdf > 获取。

22. Moore GE. Cramming more components onto integrated circuits. Electronics. 1965;38 Avaialble at <http://download.intel.com/museum/Moores_Law/Articles-Press_Releases/Gordon_Moore_1965_Article.pdf>.

23. 阮 H 编。GPU 宝石 3 .马萨诸塞州雷丁：艾迪生-韦斯利； 2008年。

23. Nguyen H, ed. GPU gems 3. Reading, MA: Addison-Wesley; 2008.

24. Nickolls J、Buck I、Garland M、Skadron K。使用 CUDA 的可扩展并行编程。ACM 队列。 2008；6(2)：40–53。

24. Nickolls J, Buck I, Garland M, Skadron K. Scalable parallel programming with CUDA. ACM Queue. 2008;6(2):40–53.

25.英伟达。CUDA 专区2012；网址为：http://www.nvidia.com/CUDA。

25. NVIDIA. CUDA Zone 2012; Available at: http://www.nvidia.com/CUDA.

26.英伟达。CUDA 编程指南 1.1 2007；位于：< http://developer.download.nvidia.com/compute/cuda/1_1/NVIDIA_CUDA_Programming_Guide_1.1.pdf >。

26. NVIDIA. CUDA Programming Guide 1.1 2007; Available at: <http://developer.download.nvidia.com/compute/cuda/1_1/NVIDIA_CUDA_Programming_Guide_1.1.pdf>.

27.英伟达。PTX：并行线程执行 ISA 版本 1.1 2007；位于：< http://www.nvidia.com/object/io_1195170102263.html >。

27. NVIDIA. PTX: Parallel Thread Execution ISA Version 1.1 2007; Available at: <http://www.nvidia.com/object/io_1195170102263.html>.

28. Nyland L、Harris M、Prins J。使用 CUDA 进行快速 N 体模拟。见：Nguyen H，编辑。GPU 宝石 3 .马萨诸塞州雷丁：艾迪生-韦斯利； 2007年。

28. Nyland L, Harris M, Prins J. Fast N-Body simulation with CUDA. In: Nguyen H, ed. GPU gems 3. Reading, MA: Addison-Wesley; 2007.

29. Oberman, SF 和 Siu, MY 高性能面积高效多功能插值器。过程。第 17 届 IEEE symp。计算机算术（第 272-279 页）。西雅图华盛顿，2005 年。

29. Oberman, S. F.and Siu,M. Y. A high-performance area-efficient multifunction interpolator. Proc. 17th IEEE symp. computer arithmetic (pp. 272–279). Seattle Washington, 2005.

30. 帕特森 DA，轩尼诗 JL。计算机组织和设计：硬件/软件接口第三版。旧金山：摩根·考夫曼； 2004年。

30. Patterson DA, Hennessy JL. Computer organization and design: The hardware/software interface 3rd ed. San Francisco: Morgan Kaufmann; 2004.

31. Pharr M，编辑。GPU Gems 2：高性能图形和通用计算的编程技术。马萨诸塞州雷丁：艾迪生-韦斯利； 2005年。

31. Pharr M, ed. GPU Gems 2: Programming techniques for high-performance graphics and general-purpose computation. Reading, MA: Addison-Wesley; 2005.

32. Satish, N. Harris, M. 和 Garland, M. 为第 23 届 IEEE 国际并行和分布式处理研讨会的议程设计高效排序算法。意大利罗马，2009 年。

32. Satish, N. Harris, M. and Garland, M. Designing efficient sorting algorithms for proceedings of the 23rd ieee international parallel and distributed processing symposium. Rome, Italy, 2009.

33. Segal M，Akeley K。opengl图形系统：A 规范，版本 2.1 2006；位于：< http://www.opengl.org/documentation/specs/ >。

33. Segal M, Akeley K. The opengl graphics system: A specification, version 2.1 2006; Available at: <http://www.opengl.org/documentation/specs/>.

34. Sengupta、S. Harris、M. Zhang、Y. 和 Owens，JD 用于 GPU 计算的扫描原语。过程。 2007 年图形硬件（第 97-106 页）。加利福尼亚州圣地亚哥，2007 年 8 月。

34. Sengupta, S. Harris, M. Zhang, Y.and Owens, J. D. Scan primitives for GPU computing. Proc. of graphics hardware 2007 (pp. 97–106). San Diego, California, Aug. 2007.

35. Hwu W，编辑。 GPU计算宝石，翡翠版。旧金山：摩根·考夫曼； 2011年。

35. Hwu W, ed. GPU computing gems, emerald edition. San Francisco: Morgan Kauffman; 2011.

36. Hwu W，编辑。 GPU计算宝石、玉石版。旧金山：摩根·考夫曼； 2011年。

36. Hwu W, ed. GPU computing gems, jade edition. San Francisco: Morgan Kauffman; 2011.

37. 斯特拉顿 JA、斯通 SS、胡 WW。 MCUDA：多核 CPU 的 CUDA 内核的高效实现。 2008 年第 21 届并行计算语言和编译器国际研讨会； [加拿大;也可作为计算机科学的讲义]。

37. Stratton JA, Stone SS, Hwu WW. MCUDA: An efficient implementation of CUDA kernels for multi-core CPUs. The 21st International Workshop on Languages and Compilers for Parallel Computing 2008; [Canada; also Available as Lecture Notes in Computer Science].

38. Volkov V.and Demmel, J. LU，使用 GPU 向量功能的 QR 和 cholesky 分解。技术报告编号 UCB/EECS-2008-49, 1–11；也可访问：< http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-49.html >。

38. Volkov V.and Demmel, J. LU, QR and cholesky factorizations using vector capabilities of GPUs. Technical Report No. UCB/EECS-2008-49, 1–11; also Available at: <http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-49.html>.

39. Williams S、Oliker L、Vuduc R、Shalf J、Yelick K、Demmel J。新兴多核平台上稀疏矩阵向量乘法的优化。Proc 超级计算 2007 (SC'07) 2007。doi 10.1145/1362622.1362674 [内华达州里诺]。

39. Williams S, Oliker L, Vuduc R, Shalf J, Yelick K, Demmel J. Optimization of sparse matrix-vector multiplication on emerging multicore platforms. Proc Supercomputing 2007 (SC’07) 2007. doi 10.1145/1362622.1362674 [Reno, Nevada].

3.1 数据并行性

3.1 Data Parallelism

现代软件应用程序通常会处理大量数据，并在顺序计算机上导致较长的执行时间。其中许多都对代表或模拟现实世界物理现象的数据进行操作。图像和视频帧是物理世界的快照，其中图片的不同部分捕获同时、独立的物理事件。刚体物理和流体动力学模拟自然力和运动，可以在小时间步长内独立评估。航空公司调度涉及并行运行的数千个航班、机组人员和机场登机口。这种独立评估是这些应用程序中数据并行性的基础。

Modern software applications often process a large amount of data and incur long execution time on sequential computers. Many of them operate on data that represents or models real-world, physical phenomena. Images and video frames are snapshots of a physical world where different parts of a picture capture simultaneous, independent physical events. Rigid-body physics and fluid dynamics model natural forces and movements that can be independently evaluated within small time steps. Airline scheduling deals with thousands of flights, crews, and airport gates that operate in parallel. Such independent evaluation is the basis of data parallelism in these applications.

让我们用图 3.1中的向量加法示例来说明数据并行性的概念。在此示例中，和向量 C 的每个元素是通过将输入向量 A 的元素与输入向量 B 的元素相加来生成的。例如，通过将 A[0] 与 B[0] 相加来生成 C[0]， C[3]是A[3]与B[3]相加生成的。所有添加都可以在平行线。因此，两个大向量的向量相加表现出丰富的数据并行性。实际应用中的数据并行可能更复杂，稍后将详细讨论。

Let us illustrate the concept of data parallelism with a vector addition example in Figure 3.1. In this example, each element of the sum vector C is generated by adding an element of input vector A to an element of input vector B. For example, C[0] is generated by adding A[0] to B[0], and C[3] is generated by adding A[3] to B[3]. All additions can be performed in parallel. Therefore, vector addition of two large vectors exhibits a rich amount of data parallelism. Data parallelism in real applications can be more complex and will be discussed in detail later.

3.2 CUDA程序结构

3.2 CUDA Program Structure

CUDA程序的结构反映了计算机中主机（CPU）和一个或多个设备（GPU）的共存。每个 CUDA 源文件都可以混合主机和设备代码。默认情况下，任何传统 C 程序都是仅包含主机代码的 CUDA 程序。可以将设备函数和数据声明添加到任何 C 源文件中。设备的函数或数据声明用特殊的 CUDA 关键字清楚地标记。这些通常是表现出大量数据并行性的函数。

The structure of a CUDA program reflects the coexistence of a host (CPU) and one or more devices (GPUs) in the computer. Each CUDA source file can have a mixture of both host and device code. By default, any traditional C program is a CUDA program that contains only host code. One can add device functions and data declarations into any C source file. The function or data declarations for the device are clearly marked with special CUDA keywords. These are typically functions that exhibit a rich amount of data parallelism.

一旦将设备函数和数据声明添加到源文件中，传统的 C 编译器就不再可接受。代码需要由能够识别并理解这些附加声明的编译器来编译。我们将使用 NVIDIA 的 CUDA C 编译器，称为 NVCC（NVIDIA C 编译器）。如图3.2顶部所示，NVCC处理CUDA程序，使用CUDA关键字来分隔主机代码和设备代码。主机代码是直接的 ANSI C 代码，使用主机的标准 C/C++ 编译器进一步编译，并作为传统 CPU 进程运行。设备代码用 CUDA 关键字标记，用于标记数据并行函数（称为内核）及其关联的数据结构。设备代码由 NVCC 的运行时组件进一步编译，在 GPU 设备上执行。在没有可用设备或内核可以在 CPU 上正确执行的情况下，人们还可以选择使用 MCUDA [Stratton 2008]等工具在 CPU 上执行内核。

Once device functions and data declarations are added to a source file, it is no longer acceptable to a traditional C compiler. The code needs to be compiled by a compiler that recognizes and understands these additional declarations. We will be using a CUDA C compiler by NVIDIA called NVCC (NVIDIA C Compiler). As shown at the top of Figure 3.2, the NVCC processes a CUDA program, using the CUDA keywords to separate the host code and device code. The host code is straight ANSI C code, which is further compiled with the host’s standard C/C++ compilers and is run as a traditional CPU process. The device code is marked with CUDA keywords for labeling data-parallel functions, called kernels, and their associated data structures. The device code is further compiled by a runtime component of NVCC and executed on a GPU device. In situations where there is no device available or a kernel can be appropriately executed on a CPU, one can also choose to execute the kernel on a CPU using tools like MCUDA [Stratton 2008].

CUDA 程序的执行如图3.3所示。执行从主机（CPU）执行开始。当调用或启动内核函数时，它由设备上的大量线程执行。内核启动生成的所有线程统称为网格。图 3.3显示了两个线程网格的执行情况。我们将很快讨论这些网格是如何组织的。当内核的所有线程完成执行时，相应的网格终止，并且执行在主机上继续，直到启动另一个内核。请注意，图 3.3显示了一个简化模型，其中 CPU 执行和 GPU 执行不重叠。许多异构计算应用程序实际上管理重叠的 CPU 和 GPU 执行，以充分利用 CPU 和 GPU。

The execution of a CUDA program is illustrated in Figure 3.3. The execution starts with host (CPU) execution. When a kernel function is called, or launched, it is executed by a large number of threads on a device. All the threads that are generated by a kernel launch are collectively called a grid. Figure 3.3 shows the execution of two grids of threads. We will discuss how these grids are organized soon. When all threads of a kernel complete their execution, the corresponding grid terminates, and the execution continues on the host until another kernel is launched. Note that Figure 3.3 shows a simplified model where the CPU execution and the GPU execution do not overlap. Many heterogeneous computing applications actually manage overlapped CPU and GPU execution to take advantage of both CPUs and GPUs.

启动内核通常会生成大量线程来利用数据并行性。在向量加法示例中，每个线程可用于计算输出向量 C 的一个元素。在这种情况下，内核将生成的线程数等于向量长度。对于长向量，会产生大量线程。由于高效的硬件支持，CUDA 程序员可以假设这些线程只需很少的时钟周期即可生成和调度。这与传统的 CPU 线程形成鲜明对比，传统的 CPU 线程通常需要数千个时钟周期来生成和调度。

Launching a kernel typically generates a large number of threads to exploit data parallelism. In the vector addition example, each thread can be used to compute one element of the output vector C. In this case, the number of threads that will be generated by the kernel is equal to the vector length. For long vectors, a large number of threads will be generated. CUDA programmers can assume that these threads take very few clock cycles to generate and schedule due to efficient hardware support. This is in contrast with traditional CPU threads that typically take thousands of clock cycles to generate and schedule.

3.3 向量加法内核

3.3 A Vector Addition Kernel

我们现在使用向量加法来说明 CUDA 编程模型。在我们展示向量加法的内核代码之前，首先回顾一下传统的仅 CPU 向量加法函数是如何工作的会很有帮助。图 3.4显示了一个简单的传统 C 程序，它由一个主函数和一个向量加法函数组成。在每一段主机代码中，我们都会为主要由主机处理的变量名称添加前缀和h_以及主要由设备d_处理的变量，以提醒我们这些变量的用途。

We now use vector addition to illustrate the CUDA programming model. Before we show the kernel code for vector addition, it is helpful to first review how a conventional CPU-only vector addition function works. Figure 3.4 shows a simple traditional C program that consists of a main function and a vector addition function. In each piece of host code, we will prefix the names of variables that are mainly processed by the host with h_ and those of variables that are mainly processed by a device d_ to remind ourselves the intended usage of these variables.

假设要相加的向量存储在主程序中分配并初始化的数组h_A和h_B中。输出向量位于数组h_C中，该数组也在主程序中初始化。为简洁起见，我们没有显示h_A、V和h_C如何分配或初始化的详细信息。附录 A提供了包含更多详细信息的完整源代码列表。指向这些数组的指针连同包含向量长度的变量N一起传递给vecAdd()函数。

Assume that the vectors to be added are stored in arrays h_A and h_B that are allocated and initialized in the main program. The output vector is in array h_C, which is also initialized in the main program. For brevity, we do not show the details of how h_A, V, and h_C are allocated or initialized. A complete source code listing that contains more details is available in Appendix A. The pointers to these arrays are passed to the vecAdd() function, along with the variable N that contains the length of the vectors.

图 3.4中的vecAdd()函数使用for循环来迭代向量元素。在第i次迭代中，输出元素C[i]接收A[i]和B[i]的和。向量长度参数n用于控制循环，使迭代次数与向量的长度相匹配。参数A、B和C通过引用传递，因此函数通过参数指针A、B和C读取h_A、h_B的元素并写入h_C的元素。当vecAdd()函数返回时，主函数中的后续语句可以访问h_C的新内容。

The vecAdd() function in Figure 3.4 uses a for loop to iterate through the vector elements. In the ith iteration, output element C[i] receives the sum of A[i] and B[i]. The vector length parameter n is used to control the loop so that the number of iterations matches the length of the vectors. The parameters A, B, and C are passed by reference so the function reads the elements of h_A, h_B and writes the elements of h_C through the parameter pointers A, B, and C. When the vecAdd() function returns, the subsequent statements in the main function can access the new contents of h_C.

并行执行向量加法的一种直接方法是修改vecAdd()函数并将其计算移至 CUDA 设备。这种修改后的vecAdd()函数的结构如图3.5所示。在文件的开头，我们需要添加 C 预处理器指令以包含CUDA.h头文件。该文件定义了我们将很快介绍的 CUDA API 函数和内置变量。该函数的第 1 部分在设备 (GPU) 内存中分配空间来保存 A、B 和 C 向量的副本，并将向量从主机内存复制到设备内存。第 2 部分在设备上启动实际向量加法内核的并行执行。第 3 部分将和向量 C 从设备内存复制回主机内存。

A straightforward way to execute vector addition in parallel is to modify the vecAdd() function and move its calculations to a CUDA device. The structure of such a modified vecAdd() function is shown in Figure 3.5. At the beginning of the file, we need to add a C preprocessor directive to include the CUDA.h header file. This file defines the CUDA API functions and built-in variables that we will be introducing soon. Part 1 of the function allocates space in the device (GPU) memory to hold copies of the A, B, and C vectors, and copies the vectors from the host memory to the device memory. Part 2 launches parallel execution of the actual vector addition kernel on the device. Part 3 copies the sum vector C from the device memory back to the host memory.

请注意，修改后的vecAdd()函数本质上是一个外包代理，它将输入数据发送到设备，激活设备上的计算，并从设备收集结果。代理这样做的方式是主程序甚至不需要知道矢量加法现在实际上是在设备上完成的。修订后的详细内容我们将在介绍 CUDA 编程模型的基本特征时介绍函数以及核函数的编写方式。

Note that the revised vecAdd() function is essentially an outsourcing agent that ships input data to a device, activates the calculation on the device, and collects the results from the device. The agent does so in such a way that the main program does not need to even be aware that the vector addition is now actually done on a device. The details of the revised function, as well as the way to compose the kernel function, will be shown as we introduce the basic features of the CUDA programming model.

3.4 器件全局内存和数据传输

3.4 Device Global Memory and Data Transfer

在CUDA中，主机和设备具有独立的内存空间。这反映了当前的现实，即设备通常是带有自己的 DRAM 的硬件卡。例如，NVIDIA GTX480 配备高达 4 GB ²（十亿字节或千兆字节）的 DRAM，称为全局内存。我们还将全局内存称为设备内存。为了在设备上执行内核，程序员需要在设备上分配全局内存，并将相关数据从主机内存传输到分配的设备内存。这对应于图 3.5的第 1 部分。类似地，在设备执行之后，编程器需要将结果数据从设备存储器传输回主机存储器，并释放不再需要的设备存储器。这对应于图 3.5的第 3 部分。 CUDA 运行时系统提供应用程序编程接口 (API) 函数来代表程序员执行这些活动。从现在开始，我们简单地说，一段数据从主机传输到设备，作为数据从主机内存复制到设备内存的简写。对于相反的方向也是如此。

In CUDA, host and devices have separate memory spaces. This reflects the current reality that devices are often hardware cards that come with their own DRAM. For example, the NVIDIA GTX480 comes with up to 4 GB² (billion bytes, or gigabytes) of DRAM, called global memory. We will also refer to global memory as device memory. To execute a kernel on a device, the programmer needs to allocate global memory on the device and transfer pertinent data from the host memory to the allocated device memory. This corresponds to Part 1 of Figure 3.5. Similarly, after device execution, the programmer needs to transfer result data from the device memory back to the host memory and free up the device memory that is no longer needed. This corresponds to Part 3 of Figure 3.5. The CUDA runtime system provides Application Programming Interface (API) functions to perform these activities on behalf of the programmer. From this point on, we will simply say that a piece of data is transferred from host to device as shorthand for saying that the data is copied from the host memory to the device memory. The same holds for the opposite direction.

图 3.6显示了 CUDA 主机内存和设备内存模型，供程序员推断设备内存的分配以及主机和设备之间的内存移动。主机可以访问设备全局存储器，以将数据传输到设备或从设备传输数据，如这些存储器和设备之间的双向箭头所示。主机如图3.6。设备内存类型比图 3.6所示的要多。常量存储器可以通过设备函数以只读方式访问，这将在第 8 章中介绍。我们还将在第 5 章中讨论寄存器和共享内存的使用。有关纹理内存的功能，请参阅CUDA 编程指南。现在，我们将重点关注全局内存的使用。

Figure 3.6 shows a CUDA host memory and device memory model for programmers to reason about the allocation of device memory and movement of memory between host and device. The device global memory can be accessed by the host to transfer data to and from the device, as illustrated by the bidirectional arrows between these memories and the host in Figure 3.6. There are more device memory types than shown in Figure 3.6. Constant memory can be accessed in a read-only manner by device functions, which will be described in Chapter 8. We will also discuss the use of registers and shared memory in Chapter 5. See the CUDA Programming Guide for the functionality of texture memory. For now, we will focus on the use of global memory.

CUDA运行时系统提供API函数来管理设备内存中的数据。例如图3.5中vecAdd()函数的第1部分和第3部分需要使用这些API函数为A、B、C分配设备内存；将 A 和 B 从主机内存传输到设备内存；将 C 从设备内存传输到主机内存；并释放A、B、C的设备内存。我们先解释一下内存分配和释放函数。图 3.7显示了两个用于分配和释放设备全局内存的 API 函数。可以从主机代码调用函数cudaMalloc()为对象分配一块设备全局内存。读者应该注意到cudaMalloc()和标准 C 运行时库malloc()之间的惊人相似之处。这是故意的； CUDA 是带有最少扩展的 C 语言。 CUDA使用标准C运行时库malloc()函数来管理主机内存，并将cudaMalloc()作为C运行时库的扩展添加。通过使接口尽可能接近原始 C 运行时库，CUDA 最大限度地减少了 C 程序员重新学习这些扩展的使用所花费的时间。

The CUDA runtime system provides API functions for managing data in the device memory. For example, Parts 1 and 3 of the vecAdd() function in Figure 3.5 need to use these API functions to allocate device memory for A, B, and C; transfer A and B from host memory to device memory; transfer C from device memory to host memory; and free the device memory for A, B, and C. We will explain the memory allocation and free functions first. Figure 3.7 shows two API functions for allocating and freeing device global memory. Function cudaMalloc() can be called from the host code to allocate a piece of device global memory for an object. Readers should notice the striking similarity between cudaMalloc() and the standard C runtime library malloc(). This is intentional; CUDA is C with minimal extensions. CUDA uses the standard C runtime library malloc() function to manage the host memory and adds cudaMalloc() as an extension to the C runtime library. By keeping the interface as close to the original C runtime libraries as possible, CUDA minimizes the time that a C programmer spends to relearn the use of these extensions.

cudaMalloc()函数的第一个参数是指针变量的地址，该变量将被设置为指向分配的对象。指针变量的地址应转换为(void ***)，因为函数需要一个通用指针；内存分配函数是一个通用函数，不限于任何特定类型的对象。³该参数允许cudaMalloc()函数将分配的内存的地址写入指针变量。⁴主机代码将此指针值传递给需要访问分配内存的内核目的。cudaMalloc()函数的第二个参数给出要分配的数据的大小（以字节为单位）。第二个参数的用法与 C malloc()函数的大小参数一致。

The first parameter to the cudaMalloc() function is the address of a pointer variable that will be set to point to the allocated object. The address of the pointer variable should be cast to (void ∗∗) because the function expects a generic pointer; the memory allocation function is a generic function that is not restricted to any particular type of objects.³ This parameter allows the cudaMalloc() function to write the address of the allocated memory into the pointer variable.⁴ The host code passes this pointer value to the kernels that need to access the allocated memory object. The second parameter to the cudaMalloc() function gives the size of the data to be allocated, in terms of bytes. The usage of this second parameter is consistent with the size parameter to the C malloc() function.

现在我们用一个简单的代码示例来说明cudaMalloc()的使用。这是图 3.5中示例的延续。为了清楚起见，我们将以d_开始一个指针变量，以指示它指向设备内存中的一个对象。程序将d_A（即&d_A）的地址转换为 void 指针后将其作为第一个参数传递。也就是说，d_A将指向为A向量分配的设备内存区域。分配区域的大小将是单精度浮点数大小的n倍，在当今大多数计算机中为 4 字节。计算完成后，以指针d_A作为输入调用cudaFree()，以从设备全局内存中释放 A 向量的存储空间。

We now use a simple code example to illustrate the use of cudaMalloc(). This is a continuation of the example in Figure 3.5. For clarity, we will start a pointer variable with d_ to indicate that it points to an object in the device memory. The program passes the address of d_A (i.e., &d_A) as the first parameter after casting it to a void pointer. That is, d_A will point to the device memory region allocated for the A vector. The size of the allocated region will be n times the size of a single-precision floating number, which is 4 bytes in most computers today. After the computation, cudaFree() is called with pointer d_A as input to free the storage space for the A vector from the device global memory.

浮动*d_A

float ∗d_A

int 大小 = n * sizeof(float);

int size = n ∗ sizeof(float);

cudaMalloc((void**)&d_A, 大小);

cudaMalloc((void∗∗)&d_A, size);

……

…

cudaFree(d_A);

cudaFree(d_A);

d_A、d_B和d_C中的地址是设备内存中的地址。这些地址不应在主机代码中取消引用。它们应该主要用于调用API函数和内核函数。在主机代码中取消引用设备内存点可能会在运行时导致异常或其他类型的运行时错误。

The addresses in d_A, d_B, and d_C are addresses in the device memory. These addresses should not be dereferenced in the host code. They should be mostly used in calling API functions and kernel functions. Dereferencing a device memory point in the host code can cause exceptions or other types of runtime errors during runtime.

读者应该使用类似的d_B和d_C指针变量声明及其相应的cudaMalloc()调用来完成图 3.5中vecAdd()示例的第 1 部分。此外，图 3.6中的第 3 部分可以通过cudaFree()调用d_B和d_C来完成。

Readers should complete Part 1 of the vecAdd() example in Figure 3.5 with similar declarations of d_B and d_C pointer variables as well as their corresponding cudaMalloc() calls. Furthermore, Part 3 in Figure 3.6 can be completed with the cudaFree() calls for d_B and d_C.

一旦主机代码为数据对象分配了设备内存，它就可以请求将数据从主机传输到设备。这是通过调用 CUDA API 函数之一来完成的。图 3.8显示了这样一个 API 函数cudaMemcpy()。 cudaMemcpy ()函数有四个参数。第一个参数是指向要复制的数据对象的目标位置的指针。第二个参数指向源位置。第三个参数指定要复制的字节数。第四个参数表示复制涉及的内存类型：从主机内存到主机内存、从主机内存到设备内存、从设备内存到主机内存、从设备内存到设备内存。例如，cudaMemcpy()函数可用于将数据从设备内存的一个位置复制到设备内存的另一位置。⁵

Once the host code has allocated device memory for the data objects, it can request that data be transferred from host to device. This is accomplished by calling one of the CUDA API functions. Figure 3.8 shows such an API function, cudaMemcpy(). The cudaMemcpy() function takes four parameters. The first parameter is a pointer to the destination location for the data object to be copied. The second parameter points to the source location. The third parameter specifies the number of bytes to be copied. The fourth parameter indicates the types of memory involved in the copy: from host memory to host memory, from host memory to device memory, from device memory to host memory, and from device memory to device memory. For example, the cudaMemcpy() function can be used to copy data from one location of the device memory to another location of the device memory.⁵

vecAdd ()函数调用cudaMemcpy()函数，在相加之前将 A 和 B 向量从主机复制到设备，并在相加完成后将 C 向量从设备复制到主机。假设A、B、d_A、d_B和size的值已经如我们之前讨论的那样设置；三个cudaMemcpy()调用如下所示。这两个符号常量cudaMemcopyHostToDevice和cudaMemcopyDeviceToHost是 CUDA 编程环境可识别的预定义常量。请注意，通过正确排序源指针和目标指针并使用适合传输类型的常量，可以使用同一函数在两个方向上传输数据。

The vecAdd() function calls the cudaMemcpy() function to copy A and B vectors from host to device before adding them and to copy the C vector from the device to host after the addition is done. Assume that the value of A, B, d_A, d_B, and size have already been set as we discussed before; the three cudaMemcpy() calls are shown below. The two symbolic constants, cudaMemcopyHostToDevice and cudaMemcopyDeviceToHost, are recognized, predefined constants of the CUDA programming environment. Note that the same function can be used to transfer data in both directions by properly ordering the source and destination pointers and using the appropriate constant for the transfer type.

cudaMemcpy(d_A, A, 大小, cudaMemcpyHostToDevice);

cudaMemcpy(d_A, A, size, cudaMemcpyHostToDevice);

cudaMemcpy(d_B, B, 大小, cudaMemcpyHostToDevice);

cudaMemcpy(d_B, B, size, cudaMemcpyHostToDevice);

cudaMemcpy(C, d_C, 大小, cudaMemcpyDeviceToHost);

cudaMemcpy(C, d_C, size, cudaMemcpyDeviceToHost);

总而言之，图 3.4中的主程序调用vecAdd()，它也在主机上执行。vecAdd()函数如图3.5所示，分配设备内存、请求数据传输并启动执行实际向量加法的内核。我们经常将这种类型的主机代码称为用于启动内核的存根函数。内核完成执行后，vecAdd()还将结果数据从设备复制到主机。我们在图 3.9中展示了vecAdd()函数的更完整版本。

To summarize, the main program in Figure 3.4 calls vecAdd(), which is also executed on the host. The vecAdd() function, outlined in Figure 3.5, allocates device memory, requests data transfers, and launches the kernel that performs the actual vector addition. We often refer to this type of host code as a stub function for launching a kernel. After the kernel finishes execution, vecAdd() also copies result data from device to the host. We show a more complete version of the vecAdd() function in Figure 3.9.

与图 3.6相比，图 3.9中的vecAdd()函数已完成第 1 部分和第 3 部分。第 1 部分为d_A、d_B和d_C分配设备内存，并将A传输到d_A，将B传输到d_B。这是通过调用cudaMalloc()和cudaMemcpy()函数来完成的。鼓励读者使用适当的参数值编写自己的函数调用，并将他们的代码与图 3.9中所示的代码进行比较。第 2 部分调用内核，将在下一节中进行描述。第 3 部分将总和数据从设备内存复制到主机内存，以便该值可供main()使用。这是通过调用cudaMemcpy()函数来完成的。然后，它从设备内存中释放d_A、d_B和d_C的内存，这是通过调用cudaFree()函数来完成的。

Compared to Figure 3.6, the vecAdd() function in Figure 3.9 is complete for Parts 1 and 3. Part 1 allocates device memory for d_A, d_B, and d_C and transfers A to d_A and B to d_B. This is done by calling the cudaMalloc() and cudaMemcpy() functions. Readers are encouraged to write their own function calls with the appropriate parameter values and compare their code with that shown in Figure 3.9. Part 2 invokes the kernel and will be described in the following section. Part 3 copies the sum data from device memory to host memory so that the value will be available to main(). This is accomplished with a call to the cudaMemcpy() function. It then frees the memory for d_A, d_B, and d_C from the device memory, which is done by calls to the cudaFree() function.

3.5 内核函数和线程

3.5 Kernel Functions and Threading

我们现在准备更多地讨论 CUDA 内核函数以及启动这些内核函数的效果。在 CUDA 中，内核函数指定在并行阶段由所有线程执行的代码。由于所有这些线程都执行相同的代码，因此 CUDA 编程是著名的 SPMD（单程序、多数据）[Atallah1998]并行编程风格的一个实例，这是大规模并行计算系统的流行编程风格。⁶

We are now ready to discuss more about the CUDA kernel functions and the effect of launching these kernel functions. In CUDA, a kernel function specifies the code to be executed by all threads during a parallel phase. Since all these threads execute the same code, CUDA programming is an instance of the well-known SPMD (single program, multiple data) [Atallah1998] parallel programming style, a popular programming style for massively parallel computing systems.⁶

当主机代码启动内核时，CUDA 运行时系统会生成以两级层次结构组织的线程网格。每个网格被组织成线程块数组，为简洁起见，将其称为块。网格中的所有块的大小相同；每个块最多可以包含 1,024 个线程。⁷ 图 3.10显示了每个块由 256 个线程组成的示例。每个线程块中的线程数是在内核启动时由主机代码指定的。同一内核可以在主机代码的不同部分启动不同数量的线程。对于给定的线程网格，块中的线程数可在blockDim变量中获得。在图3.10中， blockDim.x变量的值为256。一般来说，由于硬件效率的原因，线程块的维度应该是32的倍数。我们稍后会再讨论这个问题。

When a host code launches a kernel, the CUDA runtime system generates a grid of threads that are organized in a two-level hierarchy. Each grid is organized into an array of thread blocks, which will be referred to as blocks for brevity. All blocks of a grid are of the same size; each block can contain up to 1,024 threads.⁷ Figure 3.10 shows an example where each block consists of 256 threads. The number of threads in each thread block is specified by the host code when a kernel is launched. The same kernel can be launched with different numbers of threads at different parts of the host code. For a given grid of threads, the number of threads in a block is available in the blockDim variable. In Figure 3.10, the value of the blockDim.x variable is 256. In general, the dimensions of thread blocks should be multiples of 32 due to hardware efficiency reasons. We will revisit this later.

块中的每个线程都有一个唯一的threadIdx值。例如，块 0 中的第一个线程的threadIdx变量值为 0 ，第二个线程的值为 1，第三个线程的值为 2，等等。这允许每个线程组合其threadIdx和blockIdx值来创建唯一的全局索引对于整个网格来说。在图3.10中，数据索引i的计算方式为i = blockIdx.x*blockDim.x+threadIdx.x。由于在我们的示例中blockDim为 256，因此块 0 中线程的i值范围为 0 到 255。块 1 中线程的i值范围为 256 到 511。块 2 中线程的i值范围为 512 到 767。也就是说，这三个块中线程的i值形成从 0 到 767 的值的连续覆盖。由于每个线程使用i访问d_A、d_B和d_C，因此这些线程覆盖了原始循环的前 768 次迭代。通过启动具有更多块的内核，可以处理更大的向量。通过启动具有n或更多线程的内核，可以处理长度为n的向量。

Each thread in a block has a unique threadIdx value. For example, the first thread in block 0 has value 0 in its threadIdx variable, the second thread has value 1, the third thread has value 2, etc. This allows each thread to combine its threadIdx and blockIdx values to create a unique global index for itself with the entire grid. In Figure 3.10, a data index i is calculated as i=blockIdx.x ∗ blockDim.x + threadIdx.x. Since blockDim is 256 in our example, the i values of threads in block 0 ranges from 0 to 255. The i values of threads in block 1 range from 256 to 511. The i values of threads in block 2 range from 512 to 767. That is, the i values of the threads in these three blocks form a continuous coverage of the values from 0 to 767. Since each thread uses i to access d_A, d_B, and d_C, these threads cover the first 768 iterations of the original loop. By launching the kernel with a larger number of blocks, one can process larger vectors. By launching a kernel with n or more threads, one can process vectors of length n.

图 3.11显示了向量加法的核函数。语法是 ANSI C，带有一些值得注意的扩展。首先，在vecAddKernel()的声明前面有一个 CUDA 特定关键字__global__。该关键字指示该函数是一个内核，并且可以从主机函数调用它以在设备上生成线程网格。

Figure 3.11 shows a kernel function for a vector addition. The syntax is ANSI C with some notable extensions. First, there is a CUDA specific keyword __global__ in front of the declaration of vecAddKernel(). This keyword indicates that the function is a kernel and that it can be called from a host function to generate a grid of threads on a device.

一般来说，CUDA 在函数声明中使用三个限定符关键字来扩展 C 语言。这些关键字的含义总结在图 3.12中。__global__关键字指示所声明的函数是 CUDA 内核函数。请注意，“global”一词的两侧各有两个下划线字符。__global__函数将在设备上执行，并且只能从主机代码中调用。__device__关键字指示所声明的函数是 CUDA 设备函数。设备函数在 CUDA 设备上执行，只能从内核函数或其他设备函数调用。⁸

In general, CUDA extends C language with three qualifier keywords in function declarations. The meaning of these keywords is summarized in Figure 3.12. The __global__ keyword indicates that the function being declared is a CUDA kernel function. Note that there are two underscore characters on each side of the word “global.” A __global__ function is to be executed on the device and can only be called from the host code. The __device__ keyword indicates that the function being declared is a CUDA device function. A device function executes on a CUDA device and can only be called from a kernel function or another device function.⁸

__host__关键字指示所声明的函数是 CUDA 主机函数。主机函数只是在主机上执行的传统 C 函数，只能从另一个主机函数调用。默认情况下，如果 CUDA 程序中的所有函数在其声明中没有任何 CUDA 关键字，则它们都是主机函数。这是有道理的，因为许多 CUDA 应用程序都是从纯 CPU 执行环境移植的。程序员在移植过程中会添加内核函数和设备函数。原始功能仍作为主机功能。将所有函数默认为主机函数可以使程序员省去更改所有原始函数声明的繁琐工作。

The __host__ keyword indicates that the function being declared is a CUDA host function. A host function is simply a traditional C function that executes on the host and can only be called from another host function. By default, all functions in a CUDA program are host functions if they do not have any of the CUDA keywords in their declaration. This makes sense since many CUDA applications are ported from CPU-only execution environments. The programmer would add kernel functions and device functions during the porting process. The original functions remain as host functions. Having all functions to default into host functions spares the programmer the tedious work to change all original function declarations.

请注意，可以在函数声明中同时使用__host__和__device__ 。这种组合告诉编译系统为同一函数生成两个版本的目标文件。一种是在主机上执行的，并且只能从主机函数中调用。另一个在设备上执行，只能从设备或内核函数中调用。这支持可以重新编译相同函数源代码以生成设备版本的常见用例。许多用户库函数可能属于这一类。

Note that one can use both __host__ and __device__ in a function declaration. This combination tells the compilation system to generate two versions of object files for the same function. One is executed on the host and can only be called from a host function. The other is executed on the device and can only be called from a device or kernel function. This supports a common-use case when the same function source code can be recompiled to generate a device version. Many user library functions will likely fall into this category.

图 3.10中第二个值得注意的 ANSI C 扩展是关键字threadIdx.x。blockIdx.x和blockDim.x。请注意，所有线程都执行相同的内核代码。需要有一种方法让它们能够区分自身并将每个线程引导至数据的特定部分。这些关键字标识与为线程提供标识坐标的硬件寄存器相对应的预定义变量。不同的线程将在其threadIdx.x、blockIdx.x和blockDim.x变量中看到不同的值。为了简单起见，我们将线程称为thread _{blockIdx.x, threadIdx.x}。请注意，.x意味着可能存在.y和.z。我们很快就会回到这一点。

The second notable extension to ANSI C in Figure 3.10 is the keywords threadIdx.x. blockIdx.x, and blockDim.x. Note that all threads execute the same kernel code. There needs to be a way for them to distinguish among themselves and direct each thread toward a particular part of the data. These keywords identify predefined variables that correspond to hardware registers that provide the identifying coordinates to threads. Different threads will see different values in their threadIdx.x, blockIdx.x, and blockDim.x variables. For simplicity, we will refer to a thread as thread_{blockIdx.x, threadIdx.x}. Note that the .x implies that there might be .y and .z. We will come back to this point soon.

图 3.11中有一个自动（局部）变量i。在 CUDA 内核函数中，自动变量是每个线程私有的。也就是说，将为每个线程生成一个i版本。如果内核以 10,000 个线程启动，则i将会有 10,000 个版本，每个线程一个。线程为其i变量分配的值对其他线程不可见。我们将在第 5 章再次讨论这些自动变量。

There is an automatic (local) variable i in Figure 3.11. In a CUDA kernel function, automatic variables are private to each thread. That is, a version of i will be generated for every thread. If the kernel is launched with 10,000 threads, there will be 10,000 versions of i, one for each thread. The value assigned by a thread to its i variable is not visible to other threads. We will discuss these automatic variables again in Chapter 5.

图 3.4和图 3.11之间的快速比较揭示了 CUDA 内核和 CUDA 内核启动的重要见解。图 3.11中的核函数没有与图 3.4中的循环相对应的循环。读者应该问循环去了哪里。答案是循环现在被线程网格所取代。整个网格相当于循环。网格中的每个线程对应于原始循环的一次迭代。

A quick comparison between Figure 3.4 and Figure 3.11 reveals an important insight for CUDA kernels and a CUDA kernel launch. The kernel function in Figure 3.11 does not have a loop that corresponds to the one in Figure 3.4. Readers should ask where the loop went. The answer is that the loop is now replaced with the grid of threads. The entire grid forms the equivalent of the loop. Each thread in the grid corresponds to one iteration of the original loop.

请注意，图 3.11中的addVecKernel()中有一个if (i<n)语句。这是因为并非所有向量长度都可以表示为块大小的倍数。例如，如果向量长度为​​ 100，则最小有效线程块维度为 32。假设我们选择 32 作为块大小。需要启动 4 个线程块来处理所有 100 个向量元素。然而，四个线程块将有 128 个线程。我们需要禁止线程块 3 中的最后 28 个线程执行原始程序未预期的工作。由于所有线程都将执行相同的代码，因此所有线程都将根据n （即 100 ）测试其i值。使用if (i<n)语句，前 100 个线程将执行加法，而最后 28 个则不会。这允许内核处理任意长度的向量。

Note that there is an if (i<n) statement in addVecKernel() in Figure 3.11. This is because not all vector lengths can be expressed as multiples of the block size. For example, if the vector length is 100, the smallest efficient thread block dimension is 32. Assume that we picked 32 as the block size. One would need to launch four thread blocks to process all the 100 vector elements. However, the four thread blocks would have 128 threads. We need to disable the last 28 threads in thread block 3 from doing work not expected by the original program. Since all threads are to execute the same code, all will test their i values against n, which is 100. With the if (i<n) statement, the first 100 threads will perform the addition whereas the last 28 will not. This allows the kernel to process vectors of arbitrary lengths.

当主机代码启动内核时，它通过执行配置参数设置网格和线程块尺寸。如图 3.13所示。配置参数在传统 C 函数参数之前的<<<和>>>之间给出。第一个配置参数给出了网格中线程块的数量。第二个指定每个线程块中的线程数。在此示例中，每个块中有 256 个线程。为了确保我们有足够的线程来覆盖所有向量元素，我们将 C 上限函数应用于n/256.0。使用浮点值 256.0 确保我们为除法生成一个浮点值，以便上限值函数可以正确地将其向上舍入。例如，如果我们有 1,000 个线程，我们将启动 ceil(1,000/256.0)=4 个线程块。结果，该语句将启动 4×256=1,024 个线程。通过内核中的if (i < n)语句，如图3.11所示，前 1,000 个线程将对 1,000 个向量元素执行加法。其余 24 个不会。

When the host code launches a kernel, it sets the grid and thread block dimensions via execution configuration parameters. This is illustrated in Figure 3.13. The configuration parameters are given between the <<< and >>> before the traditional C function arguments. The first configuration parameter gives the number of thread blocks in the grid. The second specifies the number of threads in each thread block. In this example, there are 256 threads in each block. To ensure that we have enough threads to cover all the vector elements, we apply the C ceiling function to n/256.0. Using floating-point value 256.0 ensures that we generate a floating value for the division so that the ceiling function can round it up correctly. For example, if we have 1,000 threads, we would launch ceil(1,000/256.0)=4 thread blocks. As a result, the statement will launch 4×256=1,024 threads. With the if (i < n) statement in the kernel as shown in Figure 3.11, the first 1,000 threads will perform addition on the 1,000 vector elements. The remaining 24 will not.

图 3.14显示了vecAdd()的最终主机代码。该源代码完成了图 3.5中的框架。图3.10和3.14共同说明了一个简单的CUDA程序，它由主机代码和设备组成核心。该代码被硬连线为使用每个 256 个线程的线程块。但是，所使用的线程块的数量取决于向量 ( n )的长度。如果n为750，将使用三个线程块；如果n为4,000，则将使用16个线程块；如果n为 2,000,000，则将使用 7,813 个块。请注意，所有线程块都对向量的不同部分进行操作。它们可以按任意顺序执行。具有少量执行资源的小型 GPU 可以并行执行这些线程块中的一个或两个。更大的 GPU 可以并行执行 64 或 128 个块。这使得 CUDA 内核在硬件执行速度方面具有可扩展性。也就是说，相同的代码在小型 GPU 上以较低的性能运行，而在较大的 GPU 上以较高的性能运行。我们将在第 4 章中再次讨论这一点。

Figure 3.14 shows the final host code of vecAdd(). This source code completes the skeleton in Figure 3.5. Figures 3.10 and 3.14 jointly illustrate a simple CUDA program that consists of both the host code and a device kernel. The code is hardwired to use thread blocks of 256 threads each. The number of thread blocks used, however, depends on the length of the vectors (n). If n is 750, three thread blocks will be used; if n is 4,000, 16 thread blocks will be used; if n is 2,000,000, 7,813 blocks will be used. Note that all the thread blocks operate on different parts of the vectors. They can be executed in any arbitrary order. A small GPU with a small amount of execution resources may execute one or two of these thread blocks in parallel. A larger GPU may execute 64 or 128 blocks in parallel. This gives CUDA kernels scalability in execution speed with hardware. That is, same code runs at lower performance on small GPUs and higher performance on larger GPUs. We will revisit this point again in Chapter 4.

需要指出的是，使用向量加法示例是为了简单起见。实际上，分配设备内存、从主机到设备的输入数据传输、从设备到主机的输出数据传输以及取消分配设备内存的开销可能会使生成的代码比图 3.4中的原始顺序代码慢。这是因为内核完成的计算量相对于处理的数据量来说很小。对于两个浮点输入操作数和一个浮点输出操作数仅执行一次加法。实际应用程序通常具有相对于处理的数据量而言需要更多工作的内核，这使得额外的开销是值得的。他们还倾向于将数据保存在设备内存中多个内核调用，以便可以分摊开销。我们将展示此类应用的几个示例。

It is important to point out that the vector addition example is used for its simplicity. In practice, the overhead of allocating device memory, input data transfer from host to device, output data transfer from device to host, and de-allocating device memory will likely make the resulting code slower than the original sequential code in Figure 3.4. This is because the amount of calculation done by the kernel is small relative to the amount of data processed. Only one addition is performed for two floating-point input operands and one floating-point output operand. Real applications typically have kernels where much more work is needed relative to the amount of data processed, which makes the additional overhead worthwhile. They also tend to keep the data in the device memory across multiple kernel invocations so that the overhead can be amortized. We will present several examples of such applications.

3.6 总结

3.6 Summary

本章提供了 CUDA 编程模型的快速概述。 CUDA扩展了C语言以支持并行计算。我们在本章中讨论了这些扩展的一个子集。为了您的方便，我们将本章讨论的扩展总结如下。

This chapter provided a quick overview of the CUDA programming model. CUDA extends the C language to support parallel computing. We discussed a subset of these extensions in this chapter. For your convenience, we summarize the extensions that we have discussed in this chapter as follows.

3.7 练习

3.7 Exercises

参考

References

1. 阿塔拉·MJ，编辑。计算手册的算法和理论。佛罗里达州巴科拉顿：CRC Press； 1998.

1. Atallah MJ, ed. Algorithms and Theory of Computation Handbook. Baco Raton, FL: CRC Press; 1998.

2. Flynn M。一些计算机组织及其有效性。IEEE 传输计算。 1972；C-21：948。

2. Flynn M. Some computer organizations and their effectiveness. IEEE Trans Comput. 1972;C-21:948.

3. NVIDIA 公司，NVIDIA CUDA C 编程指南，版本 4.2，2012 年 4 月，可从以下位置获取：< http://developer.download.nvidia.com/compute/DevZone/docs/html/C/doc/CUDA_C_Programming_Guide.pdf >。

3. NVIDIA Corporation, NVIDIA CUDA C Programming Guide, version 4.2, April 2012, Available at: <http://developer.download.nvidia.com/compute/DevZone/docs/html/C/doc/CUDA_C_Programming_Guide.pdf>.

4.帕特·YN，帕特尔·SJ。计算系统简介：从 Bits 和 Gates 到 C 及其他 纽约：McGraw-Hill； 1972年。

4. Patt YN, Patel SJ. Introduction to Computing Systems: From Bits and Gates to C and Beyond New York: McGraw-Hill; 1972.

5. Stratton, JA, Stone, SS, & Hwu, WW MCUDA：多核 CPU 的 CUDA 内核的高效实现，第 21 届并行计算语言和编译器国际研讨会，2008 年 7 月 30-31 日，加拿大。可作为计算机科学讲义，2008 年。

5. Stratton, J. A., Stone, S. S., & Hwu, W. W. MCUDA: An Efficient Implementation of CUDA Kernels for Multi-Core CPUs, The 21st International Workshop on Languages and Compilers for Parallel Computing, July 30–31, Canada, 2008. Also available as Lecture Notes in Computer Science, 2008.

4.1 Cuda线程组织

4.1 Cuda Thread Organization

回想一下第 3 章，网格中的所有 CUDA 线程都执行相同的内核函数，它们依靠坐标来相互区分并识别要处理的数据的适当部分。这些线程被组织成两级层次结构：网格由一个或多个块组成，每个块又由一个或多个线程组成。块中的所有线程共享相同的块索引，该索引可以作为内核中的blockIdx变量进行访问。每个线程还有一个线程索引，可以作为内核中的threadIdx变量进行访问。对于 CUDA 程序员来说，blockIdx和threadIdx显示为内置的预初始化变量，可以在内核函数中访问（请参阅“内置变量”边栏）。当线程执行内核函数时，对blockIdx和threadIdx变量的引用返回线程的坐标。内核启动语句中的执行配置参数指定网格的尺寸和每个块的尺寸。这些尺寸可用作内核函数中预定义的内置变量gridDim和blockDim 。

Recall from Chapter 3 that all CUDA threads in a grid execute the same kernel function and they rely on coordinates to distinguish themselves from each other and to identify the appropriate portion of the data to process. These threads are organized into a two-level hierarchy: a grid consists of one or more blocks and each block in turn consists of one or more threads. All threads in a block share the same block index, which can be accessed as the blockIdx variable in a kernel. Each thread also has a thread index, which can be accessed as the threadIdx variable in a kernel. To a CUDA programmer, blockIdx and threadIdx appear as built-in, preinitialized variables that can be accessed within kernel functions (see “Built-in Variables” sidebar). When a thread executes a kernel function, references to the blockIdx and threadIdx variables return the coordinates of the thread. The execution configuration parameters in a kernel launch statement specify the dimensions of the grid and the dimensions of each block. These dimensions are available as predefined built-in variables gridDim and blockDim in kernel functions.

一般来说，网格是块¹的 3D 数组，每个块都是线程的 3D 数组。程序员可以通过将未使用的维度设置为 1 来选择使用更少的维度。网格的确切组织由内核启动语句的执行配置参数（在<<<和>>>内）确定。第一个执行配置参数指定网格的尺寸（以块数为单位）。第二个指定每个块的维度（以线程数表示）。每个这样的参数都是dim3类型，它是一个具有三个无符号整数字段x、y和z的 C结构体。这三个字段对应三个维度。

In general, a grid is a 3D array of blocks¹ and each block is a 3D array of threads. The programmer can choose to use fewer dimensions by setting the unused dimensions to 1. The exact organization of a grid is determined by the execution configuration parameters (within <<< and >>>) of the kernel launch statement. The first execution configuration parameter specifies the dimensions of the grid in number of blocks. The second specifies the dimensions of each block in number of threads. Each such parameter is of dim3 type, which is a C struct with three unsigned integer fields, x, y, and z. These three fields correspond to the three dimensions.

对于 1D 或 2D 网格和块，为了清楚起见，未使用的尺寸字段应设置为 1。例如，以下主机代码可用于启动vecAddkernel()内核函数并生成由 128 个块组成的一维网格，每个块由 32 个线程组成。网格中的线程总数为128×32=4,096。

For 1D or 2D grids and blocks, the unused dimension fields should be set to 1 for clarity. For example, the following host code can be used to launch the vecAddkernel() kernel function and generate a 1D grid that consists of 128 blocks, each of which consists of 32 threads. The total number of threads in the grid is 128×32=4,096.

昏暗3 昏暗块(128, 1, 1);

dim3 dimBlock(128, 1, 1);

昏暗3 昏暗网格(32, 1, 1);

dim3 dimGrid(32, 1, 1);

vecAddKernel<<<dimGrid, dimBlock>>>(…);

vecAddKernel<<<dimGrid, dimBlock>>>(…);

请注意，dimBlock和dimGrid是程序员定义的主机代码变量。这些变量可以有名称，只要它们是dim3类型并且内核启动使用适当的名称即可。例如，以下语句与前面的语句完成相同的功能：

Note that dimBlock and dimGrid are host code variables defined by the programmer. These variables can have names as long as they are of dim3 type and the kernel launch uses the appropriate names. For example, the following statements accomplish the same as the previous statements:

dim3 狗(128, 1, 1);

dim3 dog(128, 1, 1);

暗淡3猫(32,1,1);

dim3 cat(32, 1, 1);

vecAddKernel<<<狗，猫>>>(…);

vecAddKernel<<<dog, cat>>>(…);

网格和块尺寸也可以根据其他变量计算。例如图3.14中的内核启动可以写成：

The grid and block dimensions can also be calculated from other variables. For example, the kernel launch in Figure 3.14 can be written as:

dim3 dimGrid(ceil(n/256.0), 1, 1);

dim3 dimGrid(ceil(n/256.0), 1, 1);

暗淡3暗淡块(256,1,1);

dim3 dimBlock(256, 1, 1);

vecAddKernel<<<dimGrid, dimBlock>>>(…);

vecAddKernel<<<dimGrid, dimBlock>>>(…);

这允许块的数量随着向量的大小而变化，以便网格将有足够的线程来覆盖所有向量元素。内核启动时变量n的值将决定网格的维度。如果n等于 1,000，则网格将由四个块组成。如果n等于 4,000，则网格将有 16 个块。在每种情况下，都会有足够的线程来覆盖所有向量元素。一旦vecAddKernel()启动，网格和块尺寸将保持不变，直到整个网格执行完毕。

This allows the number of blocks to vary with the size of the vectors so that the grid will have enough threads to cover all vector elements. The value of variable n at kernel launch time will determine the dimension of the grid. If n is equal to 1,000, the grid will consist of four blocks. If n is equal to 4,000, the grid will have 16 blocks. In each case, there will be enough threads to cover all the vector elements. Once vecAddKernel() is launched, the grid and block dimensions will remain the same until the entire grid finishes execution.

为了方便起见，CUDA C 提供了一个特殊的快捷方式来启动具有一维网格和块的内核。除了使用dim3变量之外，还可以使用算术表达式指定一维网格和块的配置。在这种情况下，CUDA C 编译器只是将算术表达式作为x维度，并假设y和z维度为 1。这给了我们如图 3.14所示的内核启动语句：

For convenience, CUDA C provides a special shortcut for launching a kernel with 1D grids and blocks. Instead of using dim3 variables, one can use arithmetic expressions to specify the configuration of 1D grids and blocks. In this case, the CUDA C compiler simply takes the arithmetic expression as the x dimensions and assumes that the y and z dimensions are 1. This gives us the kernel launch statement shown in Figure 3.14:

 vecAddKernel<<<ceil(n/256.0), 256>>>(…);

 vecAddKernel<<<ceil(n/256.0), 256>>>(…);

在内核函数中，预定义变量gridDim和blockDim的x字段根据执行配置参数进行预初始化。例如，如果n等于 4,000，则vectAddkernel内核函数中对gridDim.x和blockDim.x 的引用将分别得到 16 和 256。请注意，与主机代码中的dim3变量不同，内核函数中这些变量的名称是 CUDA C 规范的一部分，无法更改。也就是说，核函数中的gridDim和blockDim变量始终反映网格和块的尺寸。

Within the kernel function, the x field of the predefined variables gridDim and blockDim are preinitialized according to the execution configuration parameters. For example, if n is equal to 4,000, references to gridDim.x and blockDim.x in the vectAddkernel kernel function will result in 16 and 256, respectively. Note that unlike the dim3 variables in the host code, the names of these variables within the kernel functions are part of the CUDA C specification and cannot be changed. That is, the gridDim and blockDim variables in the kernel function always reflect the dimensions of the grid and the blocks.

在 CUDA C 中， gridDim.x、gridDim.y和gridDim.z的允许值范围为 1 到 65,536。块中的所有线程共享相同的blockIdx.x、blockIdx.y和blockIdx.z值。其中，blockIdx.x 的取值范围为 0 到gridDim.x-1，blockIdx.y 的取值范围为 0 到gridDim.y-1，blockIdx.z 的取值范围为 0 到gridDim.z-1。在本书的其余部分中，我们将使用符号 ( x , y , z ) 表示 3D 网格，其中x方向上有x块，y方向上有y块，z方向上有z块。

In CUDA C, the allowed values of gridDim.x, gridDim.y, and gridDim.z range from 1 to 65,536. All threads in a block share the same blockIdx.x, blockIdx.y, and blockIdx.z values. Among all blocks, the blockIdx.x value ranges between 0 and gridDim.x-1, the blockIdx.y value between 0 and gridDim.y-1, and the blockIdx.z value between 0 and gridDim.z-1. For the rest of this book, we will use the notation (x, y, z) for a 3D grid with x blocks in the x direction, y blocks in the y direction, and z blocks in the z direction.

现在我们将注意力转向块的配置。块被组织成 3D 线程数组。可以通过将z维度设置为 1 来创建二维块。可以通过将y维度和z维度设置为 1 来创建一维块，如vectorAddkernel示例中所示。正如我们之前提到的，网格中的所有块都具有相同的尺寸。块的每个维度中的线程数由内核启动时的第二个执行配置参数指定。在内核中，此配置参数可以作为预定义变量blockDim的x、y和z字段进行访问。

We now turn our attention to the configuration of blocks. Blocks are organized into 3D arrays of threads. Two-dimensional blocks can be created by setting the z dimension to 1. One-dimensional blocks can be created by setting both the y and z dimensions to 1, as in the vectorAddkernel example. As we mentioned before, all blocks in a grid have the same dimensions. The number of threads in each dimension of a block is specified by the second execution configuration parameter at the kernel launch. Within the kernel, this configuration parameter can be accessed as the x, y, and z fields of the predefined variable blockDim.

块的总大小限制为 1,024 个线程，只要线程总数不超过 1,024，就可以灵活地将这些元素分配到三个维度。例如，(512, 1, 1)、(8, 16, 4) 和 (32, 16, 2) 都是允许的blockDim值，但 (32, 32, 2) 是不允许的，因为线程总数将超过 1,024。²

The total size of a block is limited to 1,024 threads, with flexibility in distributing these elements into the three dimensions as long as the total number of threads does not exceed 1,024. For example, (512, 1, 1), (8, 16, 4), and (32, 16, 2) are all allowable blockDim values, but (32, 32, 2) is not allowable since the total number of threads would exceed 1,024.²

请注意，网格的维度可以比块的维度更高，反之亦然。例如，图 4.1显示了一个 2D 的小玩具示例由 3D (4, 2, 2) 块组成的 (2, 2, 1) 网格。可以使用以下主机代码生成网格：

Note that the grid can have higher dimensionality than its blocks and vice versa. For example, Figure 4.1 shows a small toy example of a 2D (2, 2, 1) grid that consists of 3D (4, 2, 2) blocks. The grid can be generated with the following host code:

暗淡3暗淡块(2,2,1);

dim3 dimBlock(2, 2, 1);

昏暗3 昏暗网格(4, 2, 2);

dim3 dimGrid(4, 2, 2);

KernelFunction<<<dimGrid, dimBlock>>>(…);

KernelFunction<<<dimGrid, dimBlock>>>(…);

网格由组织成 2×2 阵列的四个块组成。图 4.1中的每个块都标有 ( blockIdx.y , blockIdx.x )。例如， block(1,0) 具有blockIdx.y=1和blockIdx.x=0。请注意，标签的顺序是最高维度排在前面。这与配置参数中使用的顺序相反，其中最低维度排在第一位。当我们说明在访问多维数组时线程坐标到数据索引的映射时，这种用于标记线程的相反顺序效果更好。

The grid consists of four blocks organized into a 2×2 array. Each block in Figure 4.1 is labeled with (blockIdx.y, blockIdx.x). For example, block(1,0) has blockIdx.y=1 and blockIdx.x=0. Note that the ordering of the labels is such that the highest dimension comes first. This is reverse of the ordering used in the configuration parameters where the lowest dimension comes first. This reversed ordering for labeling threads works better when we illustrate the mapping of thread coordinates into data indexes in accessing multidimensional arrays.

每个threadIdx还包含三个字段：x坐标threadId.x、y坐标threadIdx.y和z坐标threadIdx.z。图 4.1说明了块内线程的组织。在此示例中，每个块被组织成 4×2×2 线程阵列。由于网格内的所有块都具有相同的尺寸，因此我们只需要显示其中之一。图 4.1展开 block(1,1) 以显示其16 个线程。例如， thread(1,0,2) 具有threadIdx.z=1、threadIdx.y=0和threadIdx.x=2。请注意，在此示例中，我们有四个块，每个块有 16 个线程，网格中总共有 64 个线程。我们使用这些小数字是为了保持插图简单。典型的 CUDA 网格包含数千到数百万个线程。

Each threadIdx also consists of three fields: the x coordinate threadId.x, the y coordinate threadIdx.y, and the z coordinate threadIdx.z. Figure 4.1 illustrates the organization of threads within a block. In this example, each block is organized into 4×2×2 arrays of threads. Since all blocks within a grid have the same dimensions, we only need to show one of them. Figure 4.1 expands block(1,1) to show its 16 threads. For example, thread(1,0,2) has threadIdx.z=1, threadIdx.y=0, and threadIdx.x=2. Note that in this example, we have four blocks of 16 threads each, with a grand total of 64 threads in the grid. We use these small numbers to keep the illustration simple. Typical CUDA grids contain thousands to millions of threads.

4.2 将线程映射到多维数据

4.2 Mapping Threads to Multidimensional Data

1D、2D 或 3D 线程组织的选择通常基于数据的性质。例如，图片是像素的二维阵列。使用由 2D 块组成的 2D 网格来处理图片中的像素通常很方便。图 4.2显示了处理 76×62 图片的这种安排（水平或x方向有 76 个像素，垂直或y方向有 62 个像素）。假设我们决定使用 16×16 块，x方向有 16 个线程， y方向有 16 个线程。我们需要x方向上的 5 个块和y方向上的 4 个块，这会导致 5×4=20 个块，如图4.2所示。粗线标记了块边界。阴影区域描绘了覆盖像素的线。请注意，我们在x方向上有四个额外的线程，在y方向上有两个额外的线程。也就是说，我们将生成80×64个线程来处理76×62像素。这类似于以下情况：图 3.10中的1D vecAddKernel使用四个 256 线程块处理 1,000 元素向量。回想一下，需要一个if语句来防止额外的 24 个线程生效。类似地，我们应该期望图片处理内核函数会有if语句来测试线程索引threadIdx.x和threadIdx.y是否落在有效的像素范围内。

The choice of 1D, 2D, or 3D thread organizations is usually based on the nature of the data. For example, pictures are a 2D array of pixels. It is often convenient to use a 2D grid that consists of 2D blocks to process the pixels in a picture. Figure 4.2 shows such an arrangement for processing a 76×62 picture (76 pixels in the horizontal or x direction and 62 pixels in the vertical or y direction). Assume that we decided to use a 16×16 block, with 16 threads in the x direction and 16 threads in the y direction. We will need five blocks in the x direction and four blocks in the y direction, which results in 5×4=20 blocks as shown in Figure 4.2. The heavy lines mark the block boundaries. The shaded area depicts the threads that cover pixels. Note that we have four extra threads in the x direction and two extra threads in the y direction. That is, we will generate 80×64 threads to process 76×62 pixels. This is similar to the situation where a 1,000-element vector is processed by the 1D vecAddKernel in Figure 3.10 using four 256-thread blocks. Recall that an if statement is needed to prevent the extra 24 threads from taking effect. Analogously, we should expect that the picture processing kernel function will have if statements to test whether the thread indices threadIdx.x and threadIdx.y fall within the valid range of pixels.

假设主机代码使用整型变量n来跟踪x方向上的像素数，并使用另一个整型变量m来跟踪y方向上的像素数。我们进一步假设输入图片数据已被复制到设备内存中，并且可以通过指针变量d_Pin进行访问。输出图片已分配在设备内存中，可以通过指针变量d_Pout进行访问。以下主机代码可用于启动 2D 内核来处理图片：

Assume that the host code uses an integer variable n to track the number of pixels in the x direction, and another integer variable m to track the number of pixels in the y direction. We further assume that the input picture data has been copied to the device memory and can be accessed through a pointer variable d_Pin. The output picture has been allocated in the device memory and can be accessed through a pointer variable d_Pout. The following host code can be used to launch a 2D kernel to process the picture:

dim3 dimBlock(ceil(n/16.0), ceil(m/16.0), 1);

dim3 dimBlock(ceil(n/16.0), ceil(m/16.0), 1);

昏暗3 昏暗网格(16, 16, 1);

dim3 dimGrid(16, 16, 1);

pictureKernel<<<dimGrid, dimBlock>>>(d_Pin, d_Pout, n, m);

pictureKernel<<<dimGrid, dimBlock>>>(d_Pin, d_Pout, n, m);

在此示例中，为简单起见，我们假设块的尺寸固定为 16×16。另一方面，网格的尺寸取决于图片的尺寸。为了处理 2,000×1,500（3 M 像素）图片，我们将生成 14,100 个块，其中x方向 150 个， y方向94 个。在内核函数中，对内置变量gridDim.x、gridDim.y、blockDim.x和blockDim.y的引用将分别得到 150、94、16 和 16。

In this example, we assume for simplicity that the dimensions of the blocks are fixed at 16×16. The dimensions of the grid, on the other hand, depend on the dimensions of the picture. To process a 2,000×1,500 (3 M pixel) picture, we will generate 14,100 blocks, 150 in the x direction and 94 in the y direction. Within the kernel function, references to built-in variables gridDim.x, gridDim.y, blockDim.x, and blockDim.y will result in 150, 94, 16, and 16, respectively.

在展示内核代码之前，我们需要首先了解 C 语句如何访问动态分配的多维数组的元素。理想情况下，我们希望将d_Pin作为二维数组进行访问，其中第 j行和第i列的元素可以作为d_Pin[j][i]进行访问。然而，开发 CUDA C 所依据的 ANSI C 标准要求在编译时知道d_Pin中的列数。不幸的是，对于动态分配的数组，编译器时不知道此信息。事实上，使用动态分配数组的部分原因是允许这些数组的大小和维数在运行时根据数据大小而变化。因此，动态分配的二维数组中的列数信息在编译时是未知的。因此，程序员需要显式地将动态分配的 2D 数组线性化或“展平”为当前 CUDA C 中的等效 1D 数组。请注意，较新的 C99 标准允许动态分配的数组使用多维语法。未来的 CUDA C 版本可能会支持动态分配数组的多维语法。

Before we show the kernel code, we need to first understand how C statements access elements of dynamically allocated multidimensional arrays. Ideally, we would like to access d_Pin as a 2D array where an element at row j and column i can be accessed as d_Pin[j][i]. However, the ANSI C standard based on which CUDA C was developed requires that the number of columns in d_Pin be known at compile time. Unfortunately, this information is not known at compiler time for dynamically allocated arrays. In fact, part of the reason why one uses dynamically allocated arrays is to allow the sizes and dimensions of these arrays to vary according to data size at runtime. Thus, the information on the number of columns in a dynamically allocated 2D array is not known at compile time by design. As a result, programmers need to explicitly linearize, or “flatten,” a dynamically allocated 2D array into an equivalent 1D array in the current CUDA C. Note that the newer C99 standard allows multidimensional syntax for dynamically allocated arrays. It is likely that future CUDA C versions may support multidimensional syntax for dynamically allocated arrays.

实际上，C 中的所有多维数组都是线性化的。这是由于现代计算机中使用了“平面”内存空间（请参阅“内存空间”侧边栏）。在静态分配数组的情况下，编译器允许程序员使用更高维的索引语法（例如d_Pin[j][i]）来访问其元素。在底层，编译器将它们线性化为等效的一维数组，并将多维索引语法转换为一维偏移量。在动态分配数组的情况下，由于缺乏维度信息，当前的 CUDA C 编译器将此类转换的工作留给了程序员。

In reality, all multidimensional arrays in C are linearized. This is due to the use of a “flat” memory space in modern computers (see “Memory Space” sidebar). In the case of statically allocated arrays, the compilers allow the programmers to use higher-dimensional indexing syntax such as d_Pin[j][i] to access their elements. Under the hood, the compiler linearizes them into an equivalent 1D array and translates the multidimensional indexing syntax into a 1D offset. In the case of dynamically allocated arrays, the current CUDA C compiler leaves the work of such translation to the programmers due to lack of dimensional information.

至少有两种方法可以线性化二维数组。一种是将同一行的所有元素放置到连续的位置。然后将这些行依次放入内存空间中。这种排列称为行优先布局，如图4.3所示。为了增加可读性，我们将使用 M _j,i来表示第 j行第i列的M元素。 M _j,i相当于 C 表达式M[j][i]，但可读性稍好一些。图 4.3显示了一个示例，其中将 4×4 矩阵 M 线性化为 16 元素的一维数组，首先是第 0 行的所有元素，然后是第 1 行的四个元素，依此类推。因此，M 的一维等效索引j行i列的元素为j ×4+ i。j ×4项会跳过j行之前的行的所有元素。然后，第i项在第j行的部分中选择正确的元素。例如，M _2,1的一维索引是2×4+1=9。这如图 4.3所示，其中 M _{9是等价于 M 2,1 的}一维。这是 C 编译器线性化 2D 数组的方式。

There are at least two ways one can linearize a 2D array. One is to place all elements of the same row into consecutive locations. The rows are then placed one after another into the memory space. This arrangement, called row-major layout, is illustrated in Figure 4.3. To increase the readability, we will use M_j,i to denote an M element at the j row and the i column. M_j,i is equivalent to the C expression M[j][i] but slightly more readable. Figure 4.3 shows an example where a 4×4 matrix M is linearized into a 16-element 1D array, with all elements of row 0 first, followed by the four elements of row 1, etc. Therefore, the 1D equivalent index for the M element in row j and column i is j×4+i. The j×4 term skips over all elements of the rows before row j. The i term then selects the right element within the section for row j. For example, the 1D index for M_2,1 is 2×4+1=9. This is illustrated in Figure 4.3, where M₉ is the 1D equivalent to M_2,1. This is the way C compilers linearize 2D arrays.

线性化二维数组的另一种方法是将同一列的所有元素放置到连续的位置。然后将这些列依次放入内存空间中。这种排列称为列主布局，由 FORTRAN 编译器使用。请注意，二维数组的列优先布局等效于其转置形式的行优先布局。我们不会在这方面花费更多时间，只是要提到以前主要编程经验是使用 FORTRAN 的读者应该知道 CUDA C 使用行优先布局而不是列优先布局。此外，许多设计供 FORTRAN 程序使用的 C 库使用列优先布局来匹配 FORTRAN 编译器布局。因此，这些库的手册页，例如《基本线性代数子程序》（请参阅“线性代数函数”侧边栏），通常会告诉用户如果从 C 程序调用这些库，则需要转置输入数组。

Another way to linearize a 2D array is to place all elements of the same column into consecutive locations. The columns are then placed one after another into the memory space. This arrangement, called column-major layout, is used by FORTRAN compilers. Note that the column-major layout of a 2D array is equivalent to the row-major layout of its transposed form. We will not spend more time on this except mentioning that readers whose primary previous programming experience was with FORTRAN should be aware that CUDA C uses row-major layout rather than column-major layout. Also, many C libraries that are designed to be used by FORTRAN programs use column-major layout to match the FORTRAN compiler layout. As a result, the manual pages for these libraries, such as Basic Linear Algebra Subprograms (see “Linear Algebra Functions” sidebar), usually tell the users to transpose the input arrays if they call these libraries from C programs.

现在我们准备研究pictureKernel()的源代码，如图4.4所示。我们假设内核会将图片中的每个像素值缩放 2.0 倍。内核代码在概念上非常简单。水平方向一共有blockDim.x*gridDim.x个线程。正如我们在vecAddKernel()示例中了解到的，表达式Col = blockIdx.x*blockDim.x+threadIdx.x生成从 0 到blockDim.x*gridDim.x–1 的每个整数值。我们知道gridDim.x*blockDim.x大于或等于n。我们的线程数至少与水平方向上的像素数一样多。同样，我们也知道，至少有与垂直方向上的像素数一样多的线程。因此，只要我们测试并确保只有Row和Col值都在范围内的线程，即(Col<n) && (Row<m)，我们就能够覆盖图片中的每个像素。由于一行中有n 个像素，我们可以将行Row列Col的像素的一维索引生成为Row*n + Col。该一维索引用于读取d_Pin数组并写入d_Pout数组。

We are now ready to study the source code of pictureKernel(), shown in Figure 4.4. Let’s assume that the kernel will scale every pixel value in the picture by a factor of 2.0. The kernel code is conceptually quite simple. There are a total of blockDim.x∗gridDim.x threads in the horizontal direction. As we learned in the vecAddKernel() example, the expression Col= blockIdx.x∗blockDim.x+threadIdx.x generates every integer value from 0 to blockDim.x∗gridDim.x–1. We know that gridDim.x∗blockDim.x is greater than or equal to n. We have at least as many threads as the number of pixels in the horizontal direction. Similarly, we also know that there are at least as many threads as the number of pixels in the vertical direction. Therefore, as long as we test and make sure only the threads with both Row and Col values are within range, that is (Col<n) && (Row<m), we will be able to cover every pixel in the picture. Since there are n pixels in a row, we can generate the 1D index for the pixel at row Row and column Col as Row∗n+Col. This 1D index is used to read from the d_Pin array and write the d_Pout array.

图 4.5说明了处理 76×62 示例时pictureKernel()的执行情况。假设我们使用 16×16 块，启动pictureKernel()会生成 80×64 个线程。网格将有 20 个块，其中水平方向 5 个，垂直方向 4 个。在执行过程中，块的执行行为将填充四种不同情况之一，如图4.5所示的四个主要区域。

Figure 4.5 illustrates the execution of pictureKernel() when processing our 76×62 example. Assuming that we use 16×16 blocks, launching pictureKernel() generates 80×64 threads. The grid will have 20 blocks, 5 in the horizontal direction and 4 in the vertical direction. During the execution, the execution behavior of blocks will fill into one of four different cases, shown as four major areas in Figure 4.5.

第一个区域在图 4.5中标记为 1 ，由属于覆盖图片中大部分像素的 12 个块的线程组成。这些线程的Col和Row值都在范围内；所有这些线程都将通过if语句测试并处理图片暗阴影区域中的像素。也就是说，每个块中的所有 16×16=256 个线程都会处理像素。

The first area, marked as 1 in Figure 4.5, consists of the threads that belong to the 12 blocks covering the majority of pixels in the picture. Both Col and Row values of these threads are within range; all these threads will pass the if statement test and process pixels in the dark-shaded area of the picture. That is, all 16×16=256 threads in each block will process pixels.

第二个区域，在图 4.5中标记为 2 ，包含属于覆盖右上角的中阴影区域中的 3 个块的线程。图片的像素。尽管这些线程的Row值始终在范围内，但其中一些线程的Col值超过了n值 (76)。这是因为水平方向的线程数始终是程序员选择的blockDim.x值的倍数（本例中为 16）。覆盖 76 个像素所需的 16 的最小倍数是 80。因此，每行中的 12 个线程将在范围内找到其Col值并处理像素。另一方面，每行中有 4 个线程会发现它们的Col值超出范围，从而导致if语句条件失败。这些线程不会处理任何像素。总体而言，16×16=256 个线程中的 12×16=192 个将处理像素。

The second area, marked as 2 in Figure 4.5, contains the threads that belong to the 3 blocks in the medium-shaded area covering the upper-right pixels of the picture. Although the Row values of these threads are always within range, the Col values of some of them exceed the n value (76). This is because the number of threads in the horizontal direction is always a multiple of the blockDim.x value chosen by the programmer (16 in this case). The smallest multiple of 16 needed to cover 76 pixels is 80. As a result, 12 threads in each row will find their Col values within range and will process pixels. On the other hand, 4 threads in each row will find their Col values out of range, and thus fail the if statement condition. These threads will not process any pixels. Overall, 12×16=192 out of the 16×16=256 threads will process pixels.

第三个区域，在图 4.5中标记为 3 ，占覆盖图片中阴影区域的 3 个左下块。尽管这些线程的Col值始终在范围内，但其中一些线程的Row值超出了m值（62）。这是因为垂直方向上的线程数始终是程序员选择的blockDim.y值的倍数（本例中为 16）。 16 覆盖 62 的最小倍数是 64。因此，每列中的 14 个线程将在范围内找到其Row值并处理像素。另一方面，每列中的2个线程将导致区域2的if语句失败，并且不会处理任何像素； 256 个线程中的 16×14=224 个将处理像素。

The third area, marked as 3 in Figure 4.5, accounts for the 3 lower-left blocks covering the medium-shaded area of the picture. Although the Col values of these threads are always within range, the Row values of some of them exceed the m value (62). This is because the number of threads in the vertical direction is always multiples of the blockDim.y value chosen by the programmer (16 in this case). The smallest multiple of 16 to cover 62 is 64. As a result, 14 threads in each column will find their Row values within range and will process pixels. On the other hand, 2 threads in each column will fail the if statement of area 2, and will not process any pixels; 16×14=224 out of the 256 threads will process pixels.

第四个区域，在图 4.5中标记为 4 ，包含覆盖图片右下浅色区域的线。与区域 2 类似，前 14 行中每一行中的 4 个线程都会发现其Col值超出范围。与区域 3 类似，该块的整个底部两行都会发现其Row值超出范围。因此，16×16=256 个线程中仅允许 14×12=168 个线程处理线程。

The fourth area, marked as 4 in Figure 4.5, contains the threads that cover the lower-right light-shaded area of the picture. Similar to area 2, 4 threads in each of the top 14 rows will find their Col values out of range. Similar to area 3, the entire bottom two rows of this block will find their Row values out of range. So, only 14×12=168 of the 16×16=256 threads will be allowed to process threads.

通过在线性化数组时包含另一个维度，我们可以轻松地将 2D 数组的讨论扩展到 3D 数组。这是通过将阵列的每个“平面”依次放置来完成的。假设程序员使用变量m和n来跟踪 3D 数组中的行数和列数。程序员还需要在启动内核时确定blockDim.z和gridDim.z的值。在内核中，数组索引会涉及另一个全局索引：

We can easily extend our discussion of 2D arrays to 3D arrays by including another dimension when we linearize arrays. This is done by placing each “plane” of the array one after another. Assume that the programmer uses variables m and n to track the number of rows and columns in a 3D array. The programmer also needs to determine the values of blockDim.z and gridDim.z when launching a kernel. In the kernel, the array index will involve another global index:

int Plane = blockIdx.z*blockDim.z + threadIdx.z

int Plane = blockIdx.z∗blockDim.z + threadIdx.z

对数组P的线性化访问将采用P[Plane*m*n + Row*n+Col]的形式。当然，需要测试所有三个全局索引（Plane、Row和Col）是否都在数组的有效范围内。

The linearized access to an array P will be in the form of P[Plane∗m∗n + Row ∗ n + Col]. One would of course need to test if all the three global indices—Plane, Row, and Col—fall within the valid range of the array.

4.3 矩阵-矩阵乘法——更复杂的内核

4.3 Matrix-Matrix Multiplication—A More Complex Kernel

到目前为止，我们已经研究了vecAddkernel()和pictureKernel()，其中每个线程仅执行一次浮点算术运算一个数组元素。读者应该问一个显而易见的问题：所有 CUDA 线程都只执行这么少量的操作吗？答案是不。大多数真正的内核都让每个线程执行更多的算术运算并体现复杂的控制流。选择这两个简单的内核来教授使用threadIdx、blockIdx、blockDim和gridDim变量将线程映射到数据。特别是，我们引入了以下使用全局索引值的模式，以确保二维数组中的每个有效数据元素都由唯一的线程覆盖：

Up to this point, we have studied vecAddkernel() and pictureKernel() where each thread performs only one floating-point arithmetic operation on one array element. Readers should ask the obvious question: Do all CUDA threads perform only such a trivial amount of operation? The answer is no. Most real kernels have each thread to perform many more arithmetic operations and embody sophisticated control flows. These two simple kernels were selected for teaching the mapping of threads to data using threadIdx, blockIdx, blockDim, and gridDim variables. In particular, we introduce the following pattern of using global index values to ensure that every valid data element in a 2D array is covered by a unique thread:

行 = blockIdx.x*blockDim.x+threadIdx.x

Row = blockIdx.x∗blockDim.x+threadIdx.x

和

and

Col = blockIdx.y*blockDim.y+threadIdx.y

Col = blockIdx.y∗blockDim.y+threadIdx.y

我们还使用vecAddKernel()和pictureKernel()来引入我们创建的线程数量是块维度的倍数的现象。因此，我们最终可能会得到比数据元素更多的线程。并非所有线程都会处理数组的元素。我们使用if语句来测试线程的全局索引值是否在有效范围内。现在我们了解了线程到数据的映射，我们可以了解执行更复杂计算的内核。

We also used vecAddKernel() and pictureKernel() to introduce the phenomenon that the number of threads that we create is a multiple of the block dimension. As a result, we will likely end up with more threads than data elements. Not all threads will process elements of an array. We use an if statement to test if the global index values of a thread are within the valid range. Now that we understand the mapping of threads to data, we are in a position to understand kernels that perform more complex computation.

I×J 矩阵d_M和 J×K 矩阵d_N之间的矩阵-矩阵乘法产生 I×K 矩阵d_P。矩阵-矩阵乘法是 BLAS 标准的重要组成部分（请参阅“线性代数函数”边栏）。为简单起见，我们将讨论限制在方阵上，其中 I=J=K。我们将为I、J 和 K使用可变宽度。

Matrix–matrix multiplication between an I×J matrix d_M and a J×K matrix d_N produces an I×K matrix d_P. Matrix–matrix multiplication is an important component of the BLAS standard (see “Linear Algebra Functions” sidebar). For simplicity, we will limit our discussion to square matrices, where I=J=K. We will use variable Width for I, J, and K.

当执行矩阵-矩阵乘法时，乘积矩阵d_P的每个元素都是d_M的行和d_N的列的内积。我们将继续使用约定，其中d_P _Row,Col是Row行和Col列处的元素。如图4.6所示，d_P _{Row, Col} （ d_P中的小方块）是d_M的Row行（ d_M中的水平条）与d_N的Col列（ d_N中的垂直条）的内积）。两个向量之间的内积是对应元素的乘积之和。即，d _P _Row,Col = Σ d_M _{Row,k ∗} d_N _k,Col，对于k = 0, 1, … Width-1。例如，

When performing a matrix–matrix multiplication, each element of the product matrix d_P is an inner product of a row of d_M and a column of d_N. We will continue to use the convention where d_P_Row,Col is the element at Row row and Col column. As shown in Figure 4.6, d_P_{Row, Col} (the small square in d_P) is the inner product of the Row row of d_M (shown as the horizontal strip in d_M) and the Col column of d_N (shown as the vertical strip in d_N). The inner product between two vectors is the sum of products of corresponding elements. That is, d_P_Row,Col=∑d_M_{Row,k ∗}d_N_k,Col, for k = 0, 1, … Width-1. For example,

d_P _1,5 = d_M ₁ , ₀ *d_N _0,5 + d_M _1,1 *d_N _1,5 + d_M _1,2 * d_N _2,5 + …。 + d_M _1,宽度-1 *d_N_宽度-1,5

d_P_1,5 = d_M₁, ₀∗d_N_0,5 + d_M_1,1∗d_N_1,5 + d_M_1,2∗ d_N_2,5 +…. + d_M_1,Width-1∗d_N_Width-1,5

我们使用与pictureKernel()相同的方法将线程映射到d_P元素。即每个线程负责计算一个d_P元素。线程计算的d_P元素位于行blockIdx.y*blockDim.y+threadIdx.y和列blockIdx中。x*blockDim.x+threadIdx.x。图 4.7显示了基于线程到数据映射的内核源代码。读者应该立即看到计算Row、Col的熟悉模式，以及测试Row和Col是否都在范围内的if语句。这些语句与pictureKernel()中的对应语句几乎相同。唯一显着的区别是我们假设矩阵MulKernel()为方阵，因此我们将n和m替换为Width。

We map threads to d_P elements with the same approach as what we used for pictureKernel(). That is, each thread is responsible for calculating one d_P element. The d_P element calculated by a thread is in row blockIdx.y∗blockDim.y+threadIdx.y and in column blockIdx.x∗blockDim.x+threadIdx.x. Figure 4.7 shows the source code of the kernel based on this thread-to-data mapping. Readers should immediately see the familiar pattern of calculating Row, Col, and the if statement testing if Row and Col are both within range. These statements are almost identical to their counterparts in pictureKernel(). The only significant difference is that we are assuming square matrices for the matrixMulKernel() so we replace both n and m with Width.

通过线程到数据的映射，我们有效地将d_P划分为方形图块，其中之一如图4.6中的大方形所示。某些尺寸可能更适合一个设备，而另一些尺寸可能更适合另一个设备。这就是为什么在实际应用中，程序员通常希望将块尺寸保留为主机代码中易于调整的值。常见的做法是声明一个编译时常量，并在主机语句中使用该常量来设置内核启动配置。我们将此编译时常量称为BLOCK_WIDTH。要将BLOCK_WIDTH设置为一个值（例如 16），我们可以在头文件或使用BLOCK_WIDTH的文件开头使用以下 C 语句：

With our thread-to-data mapping, we effectively divide d_P into square tiles, one of which is shown as a large square in Figure 4.6. Some dimension sizes may be better for a device and others may be better for another device. This is why, in real applications, programmers often want to keep the block dimensions as an easily adjustable value in the host code. A common practice is to declare a compile-time constant and use this constant in the host statements for setting the kernel launch configuration. We will refer to this compile-time constant as BLOCK_WIDTH. To set BLOCK_WIDTH to a value, say 16, we can use the following C statement in a header file or the beginning of a file where BLOCK_WIDTH is used:

#定义块宽度16

#define BLOCK_WIDTH 16

在整个源代码中，程序员可以使用名称BLOCK_WIDTH，而不是使用数值。使用命名的编译时常量允许程序员在针对特定硬件进行编译时轻松地将BLOCK_WIDTH设置为不同的值。它还允许自动调整系统通过迭代地将其设置为不同的值来搜索最佳的BLOCK_WIDTH值，并针对感兴趣的硬件进行编译和运行。这种类型的过程通常称为自动调整。在这两种情况下，当改变线程块的尺寸时，源代码可以基本保持不变。

Throughout the source code, instead of using a numerical value, the programmer can use the name BLOCK_WIDTH. Using a named compile-time constant allows the programmer to easily set BLOCK_WIDTH to a different value when compiling for a particular hardware. It also allows an automated tuning system to search for the best BLOCK_WIDTH value by iteratively setting it to different values, compile, and run for the hardware of interest. This type of process is often referred to as autotuning. In both cases, the source code can remain largely unchanged while changing the dimensions of the thread blocks.

图4.8显示了用于启动matrixMulKernel()的主机代码。请注意，配置参数dimGrid 的设置是为了确保对于 Width 和BLOCK_WIDTH值的任意组合， x和y维度上都有足够的线程块来计算所有d_P元素。此外，在初始化dimGrid和dimBlock字段时使用BLOCK_WIDTH常量的名称而不是实际值。这使得程序员可以轻松更改BLOCK_WIDTH值，而无需修改任何其他语句。假设我们的宽度值为 1,000。也就是说，我们需要做 1,000×1,000 矩阵-矩阵乘法。对于BLOCK_WIDTH值为 16，我们将生成 16×16 块。网格中将有 64×64 的块来覆盖所有d_P元素。通过将图 4.8中的#define语句更改为

Figure 4.8 shows the host code to be used to launch the matrixMulKernel(). Note that the configuration parameter dimGrid is set to ensure that for any combination of Width and BLOCK_WIDTH values, there are enough thread blocks in both x and y dimensions to calculate all d_P elements. Also, the name of the BLOCK_WIDTH constant rather than the actual value is used in initializing the fields of dimGrid and dimBlock. This allows the programmer to easily change the BLOCK_WIDTH value without modifying any of the other statements. Assume that we have a Width value of 1,000. That is, we need to do 1,000×1,000 matrix–matrix multiplication. For a BLOCK_WIDTH value of 16, we will generate 16×16 blocks. There will be 64×64 blocks in the grid to cover all d_P elements. By changing the #define statement in Figure 4.8 to

#定义块宽度32

#define BLOCK_WIDTH 32

我们将生成 32×32 的块。网格中将有 32×32 块。我们可以对内核启动配置进行此更改，而无需更改任何初始化dimGrid和dimBlock的语句。

we will generate 32×32 blocks. There will be 32×32 blocks in the grid. We can make this change to the kernel launch configuration without changing any of the statements that initialize dimGrid and dimBlock.

现在我们将注意力转向每个线程所做的工作。回想一下，d_P _{Row, Col}是d_M的Row行和d_N的Col列的内积。在图 4.7中，我们使用for循环来执行此内积运算。在进入循环之前，我们将局部变量Pvalue初始化为 0。循环的每次迭代都会访问d_M的Row行中的元素和d_N的Col列中的元素，将两个元素相乘，并将乘积累加为P值。

We now turn our attention to the work done by each thread. Recall that d_P_{Row, Col} is the inner product of the Row row of d_M and the Col column of d_N. In Figure 4.7, we use a for loop to perform this inner product operation. Before we enter the loop, we initialize a local variable Pvalue to 0. Each iteration of the loop accesses an element in the Row row of d_M, an element in the Col column of d_N, multiplies the two elements together, and accumulates the product into Pvalue.

我们首先关注在for循环中访问d_M的Row行。回想一下，d_M被线性化为等效的一维数组，其中d_M的行从第 0 行开始依次放置在内存空间中。因此，第 1 行的起始元素是d_M[1*Width]，因为我们需要考虑第 0 行的所有元素。一般来说，Row行的起始元素是d_M[Row*Width]。由于一行的所有元素都放置在连续的位置，因此Row行的k元素位于d_M[Row*Width+k]处。这就是我们在图 4.7中使用的。

Let’s first focus on accessing the Row row of d_M within the for loop. Recall that d_M is linearized into an equivalent 1D array where the rows of d_M are placed one after another in the memory space, starting with the 0 row. Therefore, the beginning element of the 1 row is d_M[1∗Width] because we need to account for all elements of the 0 row. In general, the beginning element of the Row row is d_M[Row∗Width]. Since all elements of a row are placed in consecutive locations, the k element of the Row row is at d_M[Row∗Width+k]. This is what we used in Figure 4.7.

我们现在转向访问d_N的Col列。如图4.3所示， Col列的起始元素是第 0 行的Col元素，即d_N[Col]。访问Col列中的每个附加元素需要跳过整行。这是因为接下来同一列的元素实际上是下一行的相同元素。因此，Col列的第 k 个元素是d_N[k*Width+Col]。

We now turn to accessing the Col column of d_N. As shown in Figure 4.3, the beginning element of the Col column is the Col element of the 0 row, which is d_N[Col]. Accessing each additional element in the Col column requires skipping over entire rows. This is because the next element of the same column is actually the same element in the next row. Therefore, the k element of the Col column is d_N[k∗Width+Col].

执行退出for循环后，所有线程的Pvalue变量中都有其d_P元素值。然后，它使用一维等效索引表达式Row*Width+Col写入其d_P元素。同样，此索引模式与pictureKernel()中使用的类似，其中n替换为Width。

After the execution exits the for loop, all threads have their d_P element value in its Pvalue variable. It then uses the 1D equivalent index expression Row∗Width+Col to write its d_P element. Again, this index pattern is similar to that used in the pictureKernel(), with n replaced by Width.

我们现在用一个小例子来说明矩阵-矩阵乘法内核的执行。图 4.9显示了BLOCK_WIDTH = 2的4×4 d_P。较小的尺寸使我们能够将整个示例放入一张图片中。d_P矩阵现在分为四个图块，每个块计算一个图块。 （只要很明显我们正在讨论设备内存阵列，我们就会删除名称中的d_部分以提高可读性。在图 4.10中，我们使用P而不是d_P，因为很明显我们正在讨论设备内存阵列。 ）我们通过创建 2×2 线程数组块来实现这一点，每个线程计算一个P元素。在该示例中，block(0,0) 的 thread(0,0) 计算 P _0,0，而 block(1,0) 的 thread(0,0) 计算 P _2,0。很容易验证，可以通过以下公式识别由 block(1,0) 的 thread(0,0) 计算出的P元素：

We now use a small example to illustrate the execution of the matrix–matrix multiplication kernel. Figure 4.9 shows a 4×4 d_P with BLOCK_WIDTH = 2. The small sizes allow us to fit the entire example in one picture. The d_P matrix is now divided into four tiles and each block calculates one tile. (Whenever it is clear that we are discussing a device memory array, we will drop the d_ part of the name to improve readability. In Figure 4.10, we use P instead of d_P since it is clear that we are discussing a device memory array.) We do so by creating blocks that are 2×2 arrays of threads, with each thread calculating one P element. In the example, thread(0,0) of block(0,0) calculates P_0,0, whereas thread(0,0) of block(1,0) calculates P_2,0. It is easy to verify that one can identify the P element calculated by thread(0,0) of block(1,0) with the formula:

读者应该尽可能多地完成索引推导，以适应映射。

Readers should work through the index derivation for as many threads as it takes to become comfortable with the mapping.

MatrixMulKernel()中的Row和Col标识了线程要计算的P元素。 Row还标识M的行，Col标识N的列，用于线程的输入值。图 4.10说明了每个线程块中的乘法操作。对于小矩阵乘法，块(0,0)中的线程产生四个点积。block(0,0) 中 thread(0,0) 的 Row 和 Col 变量为0 * 0 + 0= 0和0*0+0=0。它映射到 P _0,0并计算 M 的第 0 行和 N 的第 0 列的点积。

Row and Col in the matrixMulKernel() identify the P element to be calculated by a thread. Row also identifies the row of M and Col identifies the column of N for input values for the thread. Figure 4.10 illustrates the multiplication actions in each thread block. For the small matrix multiplication, threads in block(0,0) produce four dot products. The Row and Col variables of thread(0,0) in block(0,0) are 0∗0+0= 0 and 0∗0+0=0. It maps to P_0,0 and calculates the dot product of row 0 of M and column 0 of N.

现在，我们将针对 block(0,0) 中的 thread(0,0)逐步执行图 4.7中的for循环。在第 0 次迭代期间（k =0），Row*Width+k = 0*4+0 = 0且k*Width+Col = 0*4+0 = 0。因此，我们正在访问d_M[0]和d_N[0]，根据图 4.3 ，它们是d_M _0,0和d_N _0,0的一维等效项。请注意，这些确实是d_M的第 0 行和d_N的第 0 列的 0 元素。

We now walk through the execution of the for loop of Figure 4.7 for thread(0,0) in block(0,0). During the 0 iteration (k=0), Row∗Width+k = 0∗4+0 = 0 and k∗Width+Col = 0∗4+0 = 0. Therefore, the we are accessing d_M[0] and d_N[0], which according to Figure 4.3 are the 1D equivalent of d_M_0,0 and d_N_0,0. Note that these are indeed the 0 elements of row 0 of d_M and column 0 of d_N.

在第一次迭代期间（k =1），Row*Width+k = 0*4+1 = 1和k*Width+Col = 1*4+0 = 4。我们正在访问d_M[1]和d_N[4]，根据图 4.3 ，它们是d_M _0,1和d_N _1,0的一维等价物。这些是d_M第0行和d_N第0列的第一个元素。

During the first iteration (k=1), Row∗Width+k = 0∗4+1 = 1 and k∗Width+Col = 1∗4+0 = 4. We are accessing d_M[1] and d_N[4], which according to Figure 4.3 are the 1D equivalent of d_M_0,1 and d_N_1,0. These are the first elements of row 0 of d_M and column 0 of d_N.

在第二次迭代期间 ( k =2)，Row*Width+k = 0*4+2 = 2和k*Width+Col = 8，结果是d_M[2]和d_N[8]。因此，访问的元素是d_M _0,2和d_N _2,0的 1D 等效项。

During the second iteration (k=2), Row∗Width+k = 0∗4+2 = 2 and k∗Width+Col = 8, which results in d_M[2] and d_N[8]. Therefore, the elements accessed are the 1D equivalent of d_M_0,2 and d_N_2,0.

最后，在第三次迭代 ( k =3) 期间，Row*Width+k = 0*4+3和k*Width+Col = 12，这导致d_M[3]和d_N[12] ，即dM的一维等价物_0,3和d_N _3,0。我们现在已经验证了for循环执行d_M的 0 行和d_N的 0 列之间的内积。循环结束后，线程写入d_P[Row*Width+Col]，即d_P[0]。这是d_P _0,0的 1D 等价物，因此 block(0,0) 中的 thread(0,0) 成功计算了d_M的 0 行和d_N的 0 列之间的内积，并将结果存入d_P _0,0。

Finally, during the third iteration (k=3), Row∗Width+k = 0∗4+3 and k∗Width+Col = 12, which results in d_M[3] and d_N[12], the 1D equivalent of dM_0,3 and d_N_3,0. We now have verified that the for loop performs the inner product between the 0 row of d_M and the 0 column of d_N. After the loop, the thread writes d_P[Row∗Width+Col], which is d_P[0]. This is the 1D equivalent of d_P_0,0 so thread(0,0) in block(0,0) successfully calculated the inner product between the 0 row of d_M and the 0 column of d_N and deposited the result in d_P_0,0.

我们将把它作为练习，让读者手动执行和验证block(0,0) 或其他块中其他线程的for循环。

We will leave it as an exercise for the reader to hand-execute and verify the for loop for other threads in block(0,0) or in other blocks.

请注意，matrixMulKernel()可以处理每个维度最多 16×65,535 个元素的矩阵。在要相乘大于此限制的矩阵的情况下，可以将P矩阵划分为尺寸可以被核覆盖的子矩阵。然后，我们可以使用主机代码迭代启动内核来完成 P 矩阵，或者使用每个线程的内核代码来计算更多 P 元素。

Note that matrixMulKernel()can handle matrices of up to 16×65,535 elements in each dimension. In the situation where matrices larger than this limit are to be multiplied, one can divide up the P matrix into submatrices of which the size can be covered by a kernel. We can then either use the host code to iteratively launch kernels to complete the P matrix or have the kernel code of each thread to calculate more P elements.

4.4 同步和透明的可扩展性

4.4 Synchronization and Transparent Scalability

到目前为止，我们已经讨论了如何启动内核以供线程网格执行。我们还解释了如何将线程映射到数据结构的某些部分。然而，我们还没有提出任何方法来协调多个线程的执行。我们现在来研究一个基本的协调机制。 CUDA 允许同一块中的线程使用屏障同步函数__syncthreads()协调其活动。请注意，“ __ ”实际上由两个“ _ ”字符组成。当内核函数调用__syncthreads()时，块中的所有线程都将保留在调用位置，直到块中的每个线程都到达该位置。这确保了块中的所有线程都已完成内核执行的一个阶段，然后它们中的任何一个都可以进入下一阶段。我们将在第 5 章中讨论__syncthreads()的一个重要用例。

So far, we have discussed how to launch a kernel for execution by a grid of threads. We have also explained how one can map threads to parts of the data structure. However, we have not yet presented any means to coordinate the execution of multiple threads. We will now study a basic coordination mechanism. CUDA allows threads in the same block to coordinate their activities using a barrier synchronization function __syncthreads(). Note that “__” actually consists of two “_” characters. When a kernel function calls __syncthreads(), all threads in a block will be held at the calling location until every thread in the block reaches the location. This ensures that all threads in a block have completed a phase of their execution of the kernel before any of them can move on to the next phase. We will discuss an important use case of __syncthreads() in Chapter 5.

屏障同步是一种简单且流行的协调并行活动的方法。在现实生活中，我们经常使用屏障同步来协调多人的并行活动。例如，假设四个朋友开车去购物中心。他们都可以去不同的商店购买自己的衣服。这是一项并行活动，并且比他们全部作为一个组并依次访问所有感兴趣的商店的情况要高效得多。然而，在他们离开商场之前，需要进行屏障同步。他们必须等到四个朋友都回到车上才能离开——先完成的需要等待后完成的。如果没有障碍同步，汽车离开时可能会将一个或多个人留在商场内，这会严重损害他们的友谊！

Barrier synchronization is a simple and popular method of coordinating parallel activities. In real life, we often use barrier synchronization to coordinate parallel activities of multiple persons. For example, assume that four friends go to a shopping mall in a car. They can all go to different stores to shop for their own clothes. This is a parallel activity and is much more efficient than if they all remain as a group and sequentially visit all the stores of interest. However, barrier synchronization is needed before they leave the mall. They have to wait until all four friends have returned to the car before they can leave—the ones who finish earlier need to wait for those who finish later. Without the barrier synchronization, one or more persons can be left in the mall when the car leaves, which can seriously damage their friendship!

图 4.11说明了屏障同步的执行过程。块中有N个线程。时间从左向右。有些线程较早到达屏障同步语句，有些则较晚。早到的人会等待迟到的人。当最后一个到达障碍物时，每个人都可以继续执行。通过屏障同步，“没有人被抛在后面。”

Figure 4.11 illustrates the execution of barrier synchronization. There are N threads in the block. Time goes from left to right. Some of the threads reach the barrier synchronization statement early and some of them much later. The ones that reach the barrier early will wait for those that arrive late. When the latest one arrives at the barrier, everyone can continue their execution. With barrier synchronization, “No one is left behind.”

在 CUDA 中，__syncthreads()语句（如果存在）必须由块中的所有线程执行。当__syncthread()语句放置在if语句中时，块中的所有线程要么执行包含__syncthreads()的路径，要么都不执行。对于if-then-else语句，如果每个路径都有__syncthreads()语句，则块中的所有线程要么执行then路径上的__syncthreads()，要么全部执行else路径。两个__syncthreads()是不同的屏障同步点。如果块中的一个线程执行then路径，另一个线程执行else路径，它们将在不同的屏障同步点等待。他们最终会永远等待对方。程序员有责任编写代码以满足这些要求。

In CUDA, a __syncthreads() statement, if present, must be executed by all threads in a block. When a __syncthread() statement is placed in an if statement, either all threads in a block execute the path that includes the __syncthreads() or none of them does. For an if-then-else statement, if each path has a __syncthreads() statement, either all threads in a block execute the __syncthreads() on the then path or all of them execute the else path. The two __syncthreads() are different barrier synchronization points. If a thread in a block executes the then path and another executes the else path, they would be waiting at different barrier synchronization points. They would end up waiting for each other forever. It is the responsibility of the programmers to write their code so that these requirements are satisfied.

同步能力还对块内的线程施加执行约束。这些线程应该在关闭时间内执行彼此距离较近，以避免等待时间过长。事实上，需要确保屏障同步中涉及的所有线程都可以访问最终到达屏障所需的资源。否则，从未到达屏障同步点的线程可能会导致其他线程永远等待。 CUDA 运行时系统通过将执行资源分配给块中的所有线程作为一个单元来满足此约束。仅当运行时系统已确保块中所有线程完成执行所需的所有资源时，块才能开始执行。当块中的一个线程被分配给执行资源时，同一块中的所有其他线程也被分配给同一资源。这确保了块中所有线程的时间接近性，并防止屏障同步期间出现过多或不确定的等待时间。

The ability to synchronize also imposes execution constraints on threads within a block. These threads should execute in close time proximity with each other to avoid excessively long waiting times. In fact, one needs to make sure that all threads involved in the barrier synchronization have access to the necessary resources to eventually arrive at the barrier. Otherwise, a thread that never arrived at the barrier synchronization point can cause everyone else to wait forever. CUDA runtime systems satisfy this constraint by assigning execution resources to all threads in a block as a unit. A block can begin execution only when the runtime system has secured all the resources needed for all threads in the block to complete execution. When a thread of a block is assigned to an execution resource, all other threads in the same block are also assigned to the same resource. This ensures the time proximity of all threads in a block and prevents excessive or indefinite waiting time during barrier synchronization.

这导致我们在 CUDA 屏障同步设计中进行重大权衡。通过不允许不同块中的线程彼此执行屏障同步，CUDA运行时系统可以以相对于彼此的任何顺序执行块，因为它们都不需要彼此等待。这种灵活性支持可扩展的实现，如图4.12所示，其中时间从上到下进行。在只有少量执行资源的低成本系统中，可以同时执行少量块；一次执行两个块如图 4.12左侧所示。在具有更多执行资源的高端实现中，可以同时执行大量块；一次执行四个块如图 4.12右侧所示。

This leads us to a major trade-off in the design of CUDA barrier synchronization. By not allowing threads in different blocks to perform barrier synchronization with each other, the CUDA runtime system can execute blocks in any order relative to each other since none of them need to wait for each other. This flexibility enables scalable implementations as shown in Figure 4.12, where time progresses from top to bottom. In a low-cost system with only a few execution resources, one can execute a small number of blocks at the same time; two blocks executing at a time is shown on the left side of Figure 4.12. In a high-end implementation with more execution resources, one can execute a large number of blocks at the same time; four blocks executing at a time is shown on the right side of Figure 4.12.

以各种速度执行相同应用程序代码的能力允许根据特定细分市场的成本、功耗和性能要求生产各种实施方案。例如，移动处理器可以缓慢但以极低的功耗执行应用程序，而桌面处理器可以以更高的速度执行相同的应用程序但消耗更多的功率。两者执行完全相同的应用程序，无需更改代码。在具有不同数量执行资源的硬件上执行相同应用程序代码的能力被称为透明可扩展性，它减轻了应用程序开发人员的负担，提高了应用程序的可用性。

The ability to execute the same application code at a wide range of speeds allows the production of a wide range of implementations according to the cost, power, and performance requirements of particular market segments. For example, a mobile processor may execute an application slowly but at extremely low power consumption, and a desktop processor may execute the same application at a higher speed while consuming more power. Both execute exactly the same application program with no change to the code. The ability to execute the same application code on hardware with a different number of execution resources is referred to as transparent scalability, which reduces the burden on application developers and improves the usability of applications.

4.5 为块分配资源

4.5 Assigning Resources to Blocks

一旦内核启动，CUDA 运行时系统就会生成相应的线程网格。正如我们在上一节中讨论的，这些线程按块分配给执行资源。在当前一代的硬件中，执行资源被组织到流式多处理器（SM）中。图 4.13说明了可以为每个 SM 分配多个线程块。每个设备对可以分配给每个 SM 的块数量都有限制。例如，CUDA 设备最多可以允许将八个块分配给每个 SM。在同时执行八个块所需的任何一种或多种类型的资源数量不足的情况下，CUDA运行时会自动减少分配给每个SM的块的数量，直到它们的组合资源使用量低于限制。由于 SM 数量有限以及可分配给每个 SM 的块数量有限，因此 CUDA 设备中可主动执行的块数量也受到限制。大多数网格包含的块比这个数量多得多。运行时系统维护需要执行的块的列表，并在 SM 完成执行先前分配给它们的块时将新块分配给 SM。

Once a kernel is launched, the CUDA runtime system generates the corresponding grid of threads. As we discussed in the previous section, these threads are assigned to execution resources on a block-by-block basis. In the current generation of hardware, the execution resources are organized into streaming multiprocessors (SMs). Figure 4.13 illustrates that multiple thread blocks can be assigned to each SM. Each device has a limit on the number of blocks that can be assigned to each SM. For example, a CUDA device may allow up to eight blocks to be assigned to each SM. In situations where there is an insufficient amount of any one or more types of resources needed for the simultaneous execution of eight blocks, the CUDA runtime automatically reduces the number of blocks assigned to each SM until their combined resource usage falls under the limit. With a limited numbers of SMs and a limited number of blocks that can be assigned to each SM, there is a limit on the number of blocks that can be actively executing in a CUDA device. Most grids contain many more blocks than this number. The runtime system maintains a list of blocks that need to execute and assigns new blocks to SMs as they complete executing the blocks previously assigned to them.

图 4.13显示了一个示例，其中为每个 SM 分配了三个线程块。 SM 资源限制之一是可以同时跟踪和调度的线程数量。 SM 需要硬件资源来维护线程和块索引并跟踪它们的执行状态。在更新的 CUDA 设备设计中，最多可以为每个 SM 分配 1,536 个线程。这可以采用 6 个块（每个块 256 个线程）、3 个块（每个块 512 个线程）等形式。如果设备在 SM 中只允许最多 8 个块，那么很明显，12 个块（每个块 128 个）每个线程都不是一个可行的选择。如果一个 CUDA 设备有 30 个 SM，每个 SM 最多可容纳 1,536 个线程，则该设备最多可以有 46,080 个线程同时驻留在 CUDA 设备中执行。

Figure 4.13 shows an example in which three thread blocks are assigned to each SM. One of the SM resource limitations is the number of threads that can be simultaneously tracked and scheduled. It takes hardware resources for SMs to maintain the thread and block indices and track their execution status. In more recent CUDA device designs, up to 1,536 threads can be assigned to each SM. This could be in the form of 6 blocks of 256 threads each, 3 blocks of 512 threads each, etc. If the device only allows up to 8 blocks in an SM, it should be obvious that 12 blocks of 128 threads each is not a viable option. If a CUDA device has 30 SMs and each SM can accommodate up to1,536 threads, the device can have up to 46,080 threads simultaneously residing in the CUDA device for execution.

4.6 查询设备属性

4.6 Querying Device Properties

我们关于向块分配执行资源的讨论提出了一个重要问题：我们如何找出可用资源的数量？当CUDA应用程序在系统上执行时，它如何找出设备中的SM数量以及可以分配给每个SM的线程数量？显然，还有其他我们尚未讨论但可能与 CUDA 应用程序的执行相关的资源。一般来说，许多现代应用程序被设计为在各种硬件系统上执行。应用程序通常需要查询底层硬件的可用资源和功能，以利用能力更强的系统，同时补偿能力较差的系统。

Our discussions on assigning execution resources to blocks raise an important question: How do we find out the amount of resources available? When a CUDA application executes on a system, how can it find out the number of SMs in a device and the number of threads that can be assigned to each SM? Obviously, there are also other resources that we have not discussed so far but can be relevant to the execution of a CUDA application. In general, many modern applications are designed to execute on a wide variety of hardware systems. There is often a need for the application to query the available resources and capabilities of the underlying hardware to take advantage of the more capable systems while compensating for the less capable systems.

在 CUDA C 中，主机代码有一个内置机制来查询系统中可用设备的属性。 CUDA运行时系统有一个API函数cudaGetDeviceCount()，它返回系统中可用的CUDA设备的数量。主机代码可以使用以下语句找出可用 CUDA 设备的数量：

In CUDA C, there is a built-in mechanism for host code to query the properties of the devices available in the system. The CUDA runtime system has an API function cudaGetDeviceCount() that returns the number of available CUDA devices in the system. The host code can find out the number of available CUDA devices using the following statements:

 int dev_count；

 int dev_count;

 cudaGetDeviceCount( &dev_count);

 cudaGetDeviceCount( &dev_count);

虽然这可能并不明显，但现代 PC 系统可以轻松拥有两个或更多 CUDA 设备。这是因为许多 PC 系统都配备了一个或多个“集成”GPU。这些 GPU 是默认图形单元，提供基本功能和硬件资源，以便为现代基于窗口的用户界面执行最少的图形功能。大多数 CUDA 应用程序在这些集成设备上的性能都不会很好。这将是主机代码迭代所有可用设备、查询其资源和功能并选择具有足够资源来执行性能令人满意的应用程序的原因。

While it may not be obvious, a modern PC system can easily have two or more CUDA devices. This is because many PC systems come with one or more “integrated” GPUs. These GPUs are the default graphics units and provide rudimentary capabilities and hardware resources to perform minimal graphics functionalities for modern window-based user interfaces. Most CUDA applications will not perform very well on these integrated devices. This would be a reason for the host code to iterate through all the available devices, query their resources and capabilities, and choose the ones that have enough resources to execute the application with satisfactory performance.

CUDA 运行时系统对系统中所有可用设备进行编号，从0到dev_count-1。它提供了一个 API 函数cudaGetDeviceProperties()，该函数返回以编号作为参数给出的设备的属性。例如，我们可以在主机代码中使用以下语句来迭代可用设备并查询其属性：

The CUDA runtime system numbers all the available devices in the system from 0 to dev_count-1. It provides an API function cudaGetDeviceProperties() that returns the properties of the device of which the number is given as an argument. For example, we can use the following statements in the host code to iterate through the available devices and query their properties:

 cudaDeviceProp dev_prop；

 cudaDeviceProp dev_prop;

 for (I = 0; i<dev_count; i++) {

 for (I = 0; i<dev_count; i++) {

  cudaGetDeviceProperties( &dev_prop, i);

  cudaGetDeviceProperties( &dev_prop, i);

  // 判断设备是否有足够的资源和能力

  // decide if device has sufficient resources and capabilities

}

}

内置类型cudaDeviceProp是一个 C 结构，其字段表示 CUDA 设备的属性。读者可以参考CUDA 编程指南来了解该类型的所有字段。我们将讨论其中一些与执行分配特别相关的字段资源到线程。我们假设属性在dev_prop变量中返回，其中的字段由cudaGetDeviceProperties()函数设置。如果读者选择以不同的方式命名变量，那么在下面的讨论中显然需要替换适当的变量名称。

The built-in type cudaDeviceProp is a C structure with fields that represent the properties of a CUDA device. Readers are referred to the CUDA Programming Guide for all the fields of the type. We will discuss a few of these fields that are particularly relevant to the assignment of execution resources to threads. We assume that the properties are returned in the dev_prop variable of which the fields are set by the cudaGetDeviceProperties() function. If readers choose to name the variable differently, the appropriate variable name will obviously need to be substituted in the following discussion.

顾名思义，字段dev_prop.maxThreadsPerBlock给出了查询设备中的块中允许的最大线程数。某些设备允许每个块中最多有 1,024 个线程，而其他设备则允许更少。未来的设备甚至可能允许每个块超过 1,024 个线程。因此，就应用程序而言，最好查询可用设备并确定哪些设备将在每个块中允许足够数量的线程。

As the name suggests, the field dev_prop.maxThreadsPerBlock gives the maximal number of threads allowed in a block in the queried device. Some devices allow up to 1,024 threads in each block and other devices allow fewer. It is possible that future devices may even allow more than 1,024 threads per block. Therefore, it is a good idea to query the available devices and determine which ones will allow a sufficient number of threads in each block as far as the application is concerned.

设备中 SM 的数量在dev_prop.multiProcessorCount中给出。正如我们之前讨论的，有些设备只有少量的 SM（例如 2 个），有些设备则有大量的 SM（例如 30 个）。如果应用程序需要大量 SM 才能达到令人满意的性能，则一定要检查预期设备的此属性。此外，设备的时钟频率位于dev_prop.clockRate中。时钟速率和 SM 数量的组合可以很好地表明设备的硬件执行能力。

The number of SMs in the device is given in dev_prop.multiProcessorCount. As we discussed earlier, some devices have only a small number of SMs (e.g., 2) and some have a much larger number of SMs (e.g., 30). If the application requires a large number of SMs to achieve satisfactory performance, it should definitely check this property of the prospective device. Furthermore, the clock frequency of the device is in dev_prop.clockRate. The combination of the clock rate and the number of SMs gives a good indication of the hardware execution capacity of the device.

主机代码可以在dev_prop.maxThreadsDim[0]（对于x维度）、dev_prop.maxThreadsDim[1]（对于y维度）和dev_prop.maxThreadsDim[2]中找到块的每个维度允许的最大线程数。 ]（对于z维度）。使用此信息的一个示例是自动调整系统在评估底层硬件的最佳性能块尺寸时设置块尺寸的范围。同样，它可以在dev_prop.maxGridSize[0]（对于x维度）、dev_prop.maxGridSize[1]（对于y维度）和dev_prop.maxGridSize[2中找到沿网格每个维度允许的最大块数。]（对于z维度）。此信息的典型用途是确定网格是否可以有足够的线程来处理整个数据集，或者是否需要某种迭代。

The host code can find the maximal number of threads allowed along each dimension of a block in dev_prop.maxThreadsDim[0] (for the x dimension), dev_prop.maxThreadsDim[1] (for the y dimension), and dev_prop.maxThreadsDim[2] (for the z dimension). An example use of this information is for an automated tuning system to set the range of block dimensions when evaluating the best performing block dimensions for the underlying hardware. Similarly, it can find the maximal number of blocks allowed along each dimension of a grid in dev_prop.maxGridSize[0] (for the x dimension), dev_prop.maxGridSize[1] (for the y dimension), and dev_prop.maxGridSize[2] (for the z dimension). A typical use of this information is to determine whether a grid can have enough threads to handle the entire data set or if some kind of iteration is needed.

cudaDeviceProp结构类型中还有更多字段。我们将在介绍它们旨在反映的概念和功能时讨论它们。

There are many more fields in the cudaDeviceProp structure type. We will discuss them as we introduce the concepts and features that they are designed to reflect.

4.7 线程调度和延迟容忍

4.7 Thread Scheduling and Latency Tolerance

线程调度严格来说是一个实现概念，因此必须在特定硬件实现的上下文中进行讨论。多数情况迄今为止的实现中，一旦将块分配给 SM，它就会被进一步划分为称为warps的 32 线程单元。扭曲的大小是特定于实现的。事实上，扭曲并不是 CUDA 规范的一部分。然而，扭曲的知识有助于理解和优化特定代 CUDA 设备上的 CUDA 应用程序的性能。 warp 的大小是 CUDA 设备的一个属性，位于设备查询变量（本例中为dev_prop ）的dev_prop.warpSize字段中。

Thread scheduling is strictly an implementation concept and thus must be discussed in the context of specific hardware implementations. In most implementations to date, once a block is assigned to a SM, it is further divided into 32-thread units called warps. The size of warps is implementation-specific. In fact, warps are not part of the CUDA specification. However, knowledge of warps can be helpful in understanding and optimizing the performance of CUDA applications on particular generations of CUDA devices. The size of warps is a property of a CUDA device, which is in the dev_prop.warpSize field of the device query variable (dev_prop in this case).

Warp是SM中线程调度的单位。图 4.14显示了实现中将块划分为扭曲的情况。每个 warp 由 32 个具有连续threadIdx值的线程组成：线程 0–31 形成第一个 warp，32–63 形成第二个 warp，依此类推。在此示例中，存在三个块：块 1、块 2 和块 3，全部分配给 SM。出于调度目的，这三个块中的每一个都被进一步划分为 warp。

The warp is the unit of thread scheduling in SMs. Figure 4.14 shows the division of blocks into warps in an implementation. Each warp consists of 32 threads of consecutive threadIdx values: threads 0–31 form the first warp, 32–63 the second warp, and so on. In this example, there are three blocks—block 1, block 2, and block 3, all assigned to an SM. Each of the three blocks is further divided into warps for scheduling purposes.

我们可以计算给定块大小和分配给每个 SM 的给定块数的 SM 中驻留的扭曲数量。例如，在图4.14中，如果每个块有256个线程，我们可以确定每个块有256÷32或8个warp。每个 SM 中有 3 个块，每个 SM 中有 8×3=24 个扭曲。

We can calculate the number of warps that reside in an SM for a given block size and a given number of blocks assigned to each SM. For example, in Figure 4.14, if each block has 256 threads, we can determine that each block has 256÷32 or 8 warps. With three blocks in each SM, we have 8×3=24 warps in each SM.

SM 被设计为遵循单指令、多数据 (SIMD) 模型来执行 warp 中的所有线程。也就是说，在任何时刻，都会为 warp 中的所有线程获取并执行一条指令。如图 4.14所示，SM 中的执行单元共享单个指令获取/调度。请注意，这些线程会将相同的指令应用于数据的不同部分。因此，warp 中的所有线程将始终具有相同的执行时序。

An SM is designed to execute all threads in a warp following the single instruction, multiple data (SIMD) model. That is, at any instant in time, one instruction is fetched and executed for all threads in the warp. This is illustrated in Figure 4.14 with a single instruction fetch/dispatch shared among execution units in the SM. Note that these threads will apply the same instruction to different portions of the data. As a result, all threads in a warp will always have the same execution timing.

图 4.14还显示了许多实际执行指令的硬件流处理器 (SP)。一般来说，SP 的数量少于分配给每个 SM 的线程数量。也就是说，每个 SM 仅具有足够的硬件来执行在任何时间点分配给 SM 的所有线程的一小部分的指令。在早期的 GPU 设计中，每个 SM 在任何给定时刻只能针对单个扭曲执行一条指令。在最近的设计中，每个 SM 都可以在任何给定时间点执行少量扭曲的指令。无论哪种情况，硬件都可以执行 SM 中所有扭曲的一小部分指令。一个合理的问题是，如果 SM 在任何时刻只能执行其中的一小部分，为什么我们需要在 SM 中拥有如此多的 warp？答案是，这就是 CUDA 处理器如何高效执行全局内存访问等长延迟操作的方式。

Figure 4.14 also shows a number of hardware streaming processors (SPs) that actually execute instructions. In general, there are fewer SPs than the number of threads assigned to each SM. That is, each SM has only enough hardware to execute instructions from a small subset of all threads assigned to the SM at any point in time. In earlier GPU design, each SM can execute only one instruction for a single warp at any given instant. In more recent designs, each SM can execute instructions for a small number of warps at any given point in time. In either case, the hardware can execute instructions for a small subset of all warps in the SM. A legitimate question is, why do we need to have so many warps in an SM if it can only execute a small subset of them at any instant? The answer is that this is how CUDA processors efficiently execute long-latency operations such as global memory accesses.

当线程束中的线程执行的指令需要等待先前启动的长延迟操作的结果时，该线程束不会被选择执行。另一个不再等待结果的常驻扭曲将被选择执行。如果多个 warp 准备好执行，则使用优先级机制来选择一个来执行。这种用其他线程的工作来填充操作延迟时间的机制通常称为延迟容忍或延迟隐藏（请参阅“延迟容忍”边栏）。

When an instruction executed by the threads in a warp needs to wait for the result of a previously initiated long-latency operation, the warp is not selected for execution. Another resident warp that is no longer waiting for results will be selected for execution. If more than one warp is ready for execution, a priority mechanism is used to select one for execution. This mechanism of filling the latency time of operations with work from other threads is often called latency tolerance or latency hiding (see “Latency Tolerance” sidebar).

请注意，warp 调度还用于容忍其他类型的操作延迟，例如流水线浮点算术和分支指令。有了足够的扭曲，硬件可能会在任何时间点找到一个扭曲来执行，从而尽管有这些长时间延迟的操作，也可以充分利用执行硬件。选择准备执行的warp不会将任何空闲时间引入执行时间线，这被称为零开销线程调度。通过 warp 调度，warp 指令的漫长等待时间通过执行其他 warp 的指令来“隐藏”。这种容忍长操作延迟的能力是 GPU 不像 CPU 那样将尽可能多的芯片面积用于缓存和分支预测机制的主要原因。因此，GPU 可以将更多的芯片面积专门用于浮点执行资源。

Note that warp scheduling is also used for tolerating other types of operation latencies such as pipelined floating-point arithmetic and branch instructions. With enough warps around, the hardware will likely find a warp to execute at any point in time, thus making full use of the execution hardware in spite of these long-latency operations. The selection of ready warps for execution does not introduce any idle time into the execution timeline, which is referred to as zero-overhead thread scheduling. With warp scheduling, the long waiting time of warp instructions is “hidden” by executing instructions from other warps. This ability to tolerate long operation latencies is the main reason why GPUs do not dedicate nearly as much chip area to cache memories and branch prediction mechanisms as CPUs. As a result, GPUs can dedicate more of its chip area to floating-point execution resources.

我们现在准备做一个简单的练习。³假设 CUDA 设备允许每个 SM 最多 8 个块和 1,024 个线程，以首先成为限制的为准。此外，它允许每个块中最多有 512 个线程。对于矩阵-矩阵乘法，我们应该使用 8×8、16×16 还是 32×32 线程块？为了回答这个问题，我们可以分析每种选择的利弊。如果我们使用 8×8 块，每个块将只有 64 个线程。我们需要 1,024÷64=12 个区块才能完全占据一个 SM。然而，由于每个 SM 中最多有 8 个块的限制，因此每个 SM 中最终只有 64×8=512 个线程。这意味着 SM 执行资源可能未得到充分利用，因为围绕长延迟操作安排的扭曲会减少。

We are now ready to do a simple exercise.³ Assume that a CUDA device allows up to 8 blocks and 1,024 threads per SM, whichever becomes a limitation first. Furthermore, it allows up to 512 threads in each block. For matrix–matrix multiplication, should we use 8×8, 16×16, or 32×32 thread blocks? To answer the question, we can analyze the pros and cons of each choice. If we use 8×8 blocks, each block would have only 64 threads. We will need 1,024÷64=12 blocks to fully occupy an SM. However, since there is a limitation of up to 8 blocks in each SM, we will end up with only 64×8=512 threads in each SM. This means that the SM execution resources will likely be underutilized because there will be fewer warps to schedule around long-latency operations.

16×16 块为每个块提供 256 个线程。这意味着每个 SM 可以占用 1,024÷256=4 个区块。这在 8 块限制之内。这是一个很好的配置，因为我们将在每个 SM 中拥有完整的线程容量以及用于围绕长延迟操作进行调度的最大数量的扭曲。 32×32 块将在每个块中提供 1,024 个线程，超出了该设备每块 512 个线程的限制。

The 16×16 blocks give 256 threads per block. This means that each SM can take 1,024÷256=4 blocks. This is within the 8-block limitation. This is a good configuration since we will have full thread capacity in each SM and a maximal number of warps for scheduling around the long-latency operations. The 32×32 blocks would give 1,024 threads in each block, exceeding the limit of 512 threads per block for this device.

4.8 总结

4.8 Summary

内核执行配置定义了网格及其块的维度。blockIdx和threadIdx变量中的唯一坐标允许网格线程识别自身及其数据域。程序员有责任在内核函数中使用这些变量，以便线程可以正确识别要处理的数据部分。这种编程模型迫使程序员将线程及其数据组织成分层和多维的组织。

The kernel execution configuration defines the dimensions of a grid and its blocks. Unique coordinates in blockIdx and threadIdx variables allow threads of a grid to identify themselves and their domains of data. It is the programmer’s responsibility to use these variables in kernel functions so that the threads can properly identify the portion of the data to process. This model of programming compels the programmer to organize threads and their data into hierarchical and multidimensional organizations.

一旦网格启动，其块就会以任意顺序分配给 SM，从而实现 CUDA 应用程序的透明可扩展性。透明的可扩展性有一个限制：不同块中的线程无法彼此同步。为了让内核保持透明的可伸缩性，不同块中的线程相互同步的简单方法是终止内核并为同步点之后的活动启动新的内核。

Once a grid is launched, its blocks are assigned to SMs in arbitrary order, resulting in transparent scalability of CUDA applications. The transparent scalability comes with a limitation: threads in different blocks cannot synchronize with each other. To allow a kernel to maintain transparent scalability, the simple way for threads in different blocks to synchronize with each other is to terminate the kernel and start a new kernel for the activities after the synchronization point.

线程被分配给 SM 来逐块执行。每个 CUDA 设备对每个 SM 中的可用资源量施加潜在不同的限制。例如，每个 CUDA 设备对其线程块数量和每个 SM 可容纳的线程数量都有限制，以首先成为限制的为准。对于每个内核，这些资源限制中的一个或多个可能成为同时驻留在 CUDA 设备中的线程数量的限制因素。

Threads are assigned to SMs for execution on a block-by-block basis. Each CUDA device imposes a potentially different limitation on the amount of resources available in each SM. For example, each CUDA device has a limit on the number of thread blocks and the number of threads each of its SMs can accommodate, whichever becomes a limitation first. For each kernel, one or more of these resource limitations can become the limiting factor for the number of threads that simultaneously reside in a CUDA device.

一旦一个块被分配给 SM，它就会被进一步划分为 warp。 warp 中的所有线程都具有相同的执行时序。在任何时候，SM 都只执行其常驻扭曲的一小部分指令。这允许其他线程等待长延迟操作，而不会减慢大量执行单元的整体执行吞吐量。

Once a block is assigned to an SM, it is further partitioned into warps. All threads in a warp have identical execution timing. At any time, the SM executes instructions of only a small subset of its resident warps. This allows the other warps to wait for long-latency operations without slowing down the overall execution throughput of the massive number of execution units.

4.9 练习

4.9 Exercises

5.1 内存访问效率的重要性

5.1 Importance of Memory Access Efficiency

我们可以通过计算图 4.7中矩阵乘法内核代码的预期性能水平（如图5.1所示）来说明内存访问效率的影响。就执行时间而言，内核中最重要的部分是执行内积计算的for循环。

We can illustrate the effect of memory access efficiency by calculating the expected performance level of the matrix multiplication kernel code in Figure 4.7, replicated in Figure 5.1. The most important part of the kernel in terms of execution time is the for loop that performs inner product calculation.

for (int k = 0; k < 宽度; ++k)

for (int k = 0; k < Width; ++k)

 P值+= d_M[行*宽度+k] * d_N[k*宽度+列];

 Pvalue += d_M[Row∗Width+k] ∗ d_N[k∗Width+Col];

在此循环的每次迭代中，都会执行两次全局内存访问，以进行一次浮点乘法和一次浮点加法。一个全局内存访问获取d_M[]元素，另一个全局内存访问获取d_N[]元素。一个浮点运算将获取的d_M[]和d_N[]元素相乘，另一个浮点运算将乘积累加到Pvalue中。因此，浮点计算与全局内存访问操作的比例为1:1，即1.0。我们将此比率称为计算与全局内存访问 (CGMA) 比率，定义为每次访问 CUDA 程序区域内的全局内存时执行的浮点计算数量。

In every iteration of this loop, two global memory accesses are performed for one floating-point multiplication and one floating-point addition. One global memory access fetches a d_M[] element and the other fetches a d_N[] element. One floating-point operation multiplies the d_M[] and d_N[] elements fetched and the other accumulates the product into Pvalue. Thus, the ratio of floating-point calculation to global memory access operation is 1:1, or 1.0. We will refer to this ratio as the compute to global memory access (CGMA) ratio, defined as the number of floating-point calculations performed for each access to the global memory within a region of a CUDA program.

CGMA 对 CUDA 内核的性能有重大影响。在当今的高端设备中，全局内存带宽约为 200 GB/s。每个单精度浮点值有 4 个字节，预计每秒加载的单精度操作数不会超过 50 (200/4) 千兆。 CGMA 比率为 1.0 时，矩阵乘法内核每秒执行不超过 50 次千兆浮点运算 (GFLOPS)。虽然 50 GFLOPS 是一个值得尊敬的数字，但对于这些高端设备来说，它只是 1,500 GFLOPS 或更高的峰值单精度性能的一小部分。我们需要增加 CGMA 比率来为内核实现更高水平的性能。为了使矩阵乘法代码达到处理器的峰值 1,500 GFLOPS 额定值，我们需要 CGMA 值为 30。在过去三代设备中，所需的 CGMA 比率大约增加了一倍。

CGMA has major implications on the performance of a CUDA kernel. In a high-end device today, the global memory bandwidth is around 200 GB/s. With 4 bytes in each single-precision floating-point value, one can expect to load no more than 50 (200/4) giga single-precision operands per second. With a CGMA ration of 1.0, the matrix multiplication kernel will execute no more than 50 giga floating-point operations per second (GFLOPS). While 50 GFLOPS is a respectable number, it is only a tiny fraction of the peak single-precision performance of 1,500 GFLOPS or higher for these high-end devices. We need to increase the CGMA ratio to achieve a higher level of performance for the kernel. For the matrix multiplication code to achieve the peak 1,500 GFLOPS rating of the processor, we need a CGMA value of 30. The desired CGMA ratio has roughly doubled in the past three generations of devices.

5.2 CUDA 设备内存类型

5.2 CUDA Device Memory Types

CUDA 支持多种类型的内存，程序员可以使用这些内存来实现高 CGMA 比率，从而在内核中实现高执行速度。图 5.2显示了这些 CUDA 设备存储器。在图的底部，我们看到全局内存和常量内存。主机可以通过调用 API 函数对这些类型的存储器进行写入 (W) 和读取 (R)。¹我们已经在第 3 章中介绍了全局内存。当所有线程同时访问同一位置时，恒定内存支持设备的短延迟、高带宽、只读访问。

CUDA supports several types of memory that can be used by programmers to achieve a high CGMA ratio and thus a high execution speed in their kernels. Figure 5.2 shows these CUDA device memories. At the bottom of the figure, we see global memory and constant memory. These types of memory can be written (W) and read (R) by the host by calling API functions.¹ We have already introduced global memory in Chapter 3. The constant memory supports short-latency, high-bandwidth, read-only access by the device when all threads simultaneously access the same location.

图5.2中的寄存器和共享存储器是片上存储器。可以以高度并行的方式以非常高的速度访问驻留在这些类型的存储器中的变量。寄存器被分配给各个线程；每个线程只能访问自己的寄存器。内核函数通常使用寄存器来保存经常访问的变量每个线程私有。共享内存分配给线程块；块中的所有线程都可以访问分配给该块的共享内存位置中的变量。共享内存是线程通过共享输入数据和工作中间结果进行协作的有效手段。通过在一种 CUDA 内存类型中声明 CUDA 变量，CUDA 程序员可以规定该变量的可见性和访问速度。

Registers and shared memory in Figure 5.2 are on-chip memories. Variables that reside in these types of memory can be accessed at very high speed in a highly parallel manner. Registers are allocated to individual threads; each thread can only access its own registers. A kernel function typically uses registers to hold frequently accessed variables that are private to each thread. Shared memory is allocated to thread blocks; all threads in a block can access variables in the shared memory locations allocated to the block. Shared memory is an efficient means for threads to cooperate by sharing their input data and the intermediate results of their work. By declaring a CUDA variable in one of the CUDA memory types, a CUDA programmer dictates the visibility and access speed of the variable.

为了充分理解寄存器、共享内存和全局内存之间的区别，我们需要更详细地了解这些不同类型的内存在现代处理器中是如何实现和使用的。 CUDA 编程模型中的全局内存映射到冯诺依曼模型的内存（请参阅“冯诺依曼模型”边栏）。图 5.3中的处理器框对应于我们今天通常看到的处理器芯片边界。全局存储器脱离处理器芯片并采用 DRAM 技术实现，这意味着较长的访问延迟和相对较低的访问带宽。这些寄存器对应于冯诺依曼模型的“寄存器文件”。它位于处理器芯片上，这意味着非常短的访问延迟和显着更高的访问带宽。在典型的设备中，寄存器文件的聚合访问带宽大约是全局存储器的两个数量级。此外，每当变量存储在寄存器中时，其访问不再消耗片外全局存储器带宽。这将反映为 CGMA 比率的增加。

To fully appreciate the difference between registers, shared memory, and global memory, we need to go into a little more detail of how these different types of memories are realized and used in modern processors. The global memory in the CUDA programming model maps to the memory of the von Neumann model (see “The von Neumann Model” sidebar). The processor box in Figure 5.3 corresponds to the processor chip boundary that we typically see today. The global memory is off the processor chip and is implemented with DRAM technology, which implies long access latencies and relatively low access bandwidth. The registers correspond to the “register file” of the von Neumann model. It is on the processor chip, which implies very short access latency and drastically higher access bandwidth. In a typical device, the aggregated access bandwidth of the register files is about two orders of magnitude of that of the global memory. Furthermore, whenever a variable is stored in a register, its accesses no longer consume off-chip global memory bandwidth. This will be reflected as an increase in the CGMA ratio.

更微妙的一点是，每次访问寄存器所涉及的指令比全局内存少。在图 5.3中，处理器使用 PC 值将指令从内存提取到 IR 中（请参阅“冯·诺依曼模型”边栏）。然后，所获取的指令的位被用来控制计算机组件的活动。使用指令位来控制计算机的活动称为指令执行。每个时钟周期内可以获取和执行的指令数量是有限的。因此，程序需要执行的指令越多，执行程序所需的时间就越多。

A more subtle point is that each access to registers involves fewer instructions than global memory. In Figure 5.3, the processor uses the PC value to fetch instructions from memory into the IR (see “The von Neumann Model” sidebar). The bits of the fetched instructions are then used to control the activities of the components of the computer. Using the instruction bits to control the activities of the computer is referred to as instruction execution. The number of instructions that can be fetched and executed in each clock cycle is limited. Therefore, the more instructions that need to be executed for a program, the more time it can take to execute the program.

大多数现代处理器中的算术指令都有“内置”寄存器操作数。例如，典型的浮点加法指令的形式为

Arithmetic instructions in most modern processors have “built-in” register operands. For example, a typical floating addition instruction is of the form

 时尚 r1、r2、r3

 fadd r1, r2, r3

其中r2和r3是寄存器编号，指定寄存器文件中可以找到输入操作数值的位置。浮点加法结果值的存储位置由r1指定。因此，当算术指令的操作数位于寄存器中时，不需要额外的指令来使操作数值可供完成算术计算的算术和逻辑单元（ALU）使用。

where r2 and r3 are the register numbers that specify the location in the register file where the input operand values can be found. The location for storing the floating-point addition result value is specified by r1. Therefore, when an operand of an arithmetic instruction is in a register, there is no additional instruction required to make the operand value available to the arithmetic and logic unit (ALU) where the arithmetic calculation is done.

另一方面，如果操作数值位于全局存储器中，则需要执行存储器加载操作以使操作数值可供ALU使用。例如，如果浮点加法指令的第一个操作数位于当今典型计算机的全局存储器中，则所涉及的指令可能是

On the other hand, if an operand value is in global memory, one needs to perform a memory load operation to make the operand value available to the ALU. For example, if the first operand of a floating-point addition instruction is in global memory of a typical computer today, the instructions involved will likely be

 负载 r2、r4、偏移

 load r2, r4, offset

 时尚 r1、r2、r3

 fadd r1, r2, r3

其中加载指令将偏移值添加到r4的内容以形成操作数值的地址。然后它访问全局内存并将值放入寄存器r2中。fadd指令然后使用r2和r3中的值执行浮点加法，并将结果放入r1中。由于处理器每个时钟周期只能获取和执行有限数量的指令，因此具有额外负载的版本可能会比没有负载的版本花费更多的时间来处理。这也是为什么将操作数放在寄存器中可以提高执行速度的另一个原因。

where the load instruction adds an offset value to the contents of r4 to form an address for the operand value. It then accesses the global memory and places the value into register r2. The fadd instruction then performs the floating addition using the values in r2 and r3 and places the result into r1. Since the processor can only fetch and execute a limited number of instructions per clock cycle, the version with an additional load will likely take more time to process than the one without. This is another reason why placing the operands in registers can improve execution speed.

最后，还有另一个微妙的原因为什么最好将操作数值放在寄存器中。在现代计算机中，从寄存器堆访问值所消耗的能量比从全局存储器访问值所消耗的能量至少低一个数量级。我们很快就会了解现代计算机中访问这两种硬件结构的速度和能量差异的更多细节。然而，我们很快就会了解到，在当今的 GPU 中，每个线程可用的寄存器数量非常有限。我们需要小心，不要过度订阅这一有限的资源。

Finally, there is another subtle reason why placing an operand value in registers is preferable. In modern computers, the energy consumed for accessing a value from the register file is at least an order of magnitude lower than for accessing a value from the global memory. We will look at more details of the speed and energy difference in accessing these two hardware structures in modern computers soon. However, as we will soon learn, the number of registers available to each thread is quite limited in today’s GPUs. We need to be careful not to oversubscribe to this limited resource.

图 5.4显示了 CUDA 设备中的共享内存和寄存器。尽管两者都是片上存储器，但它们在功能和访问成本方面存在显着差异。共享内存被设计为驻留在处理器芯片上的内存空间的一部分（参见第 4.2 节）。当处理器访问驻留在共享内存中的数据时，它需要执行内存加载操作，就像访问全局内存中的数据一样。然而，由于共享内存驻留在片上，因此与全局内存相比，它的访问延迟要低得多，带宽要高得多。由于需要执行加载操作，共享内存比寄存器具有更长的延迟和更低的带宽。在计算机体系结构中，共享内存是暂存器内存的一种形式。

Figure 5.4 shows shared memory and registers in a CUDA device. Although both are on-chip memories, they differ significantly in functionality and cost of access. Shared memory is designed as part of the memory space that resides on the processor chip (see Section 4.2). When the processor accesses data that resides in the shared memory, it needs to perform a memory load operation, just like accessing data in the global memory. However, because shared memory resides on-chip, it can be accessed with much lower latency and much higher bandwidth than the global memory. Because of the need to perform a load operation, share memory has longer latency and lower bandwidth than registers. In computer architecture, share memory is a form of scratchpad memory.

CUDA 中的共享内存和寄存器之间的一个重要区别是，驻留在共享内存中的变量可由块中的所有线程访问。这与寄存器数据相反，寄存器数据是线程私有的。也就是说，共享内存旨在支持块中线程之间高效、高带宽的数据共享。如图5.4所示，CUDA 设备 SM 通常采用多个处理单元（在图 4.14中称为 SP），以允许多个线程同时进行（请参阅“处理单元和线程”边栏）。块中的线程可以分布在这些处理单元上。因此，这些 CUDA 设备中共享内存的硬件实现通常被设计为允许多个处理单元同时访问其内容，以支持块中线程之间的高效数据共享。我们将学习几种重要类型的并行算法，这些算法可以从线程之间有效的数据共享中受益匪浅。

One important difference between the share memory and registers in CUDA is that variables that reside in the shared memory are accessible by all threads in a block. This is in contrast to register data, which is private to a thread. That is, shared memory is designed to support efficient, high-bandwidth sharing of data among threads in a block. As shown in Figure 5.4, a CUDA device SM typically employs multiple processing units, referred to as SPs in Figure 4.14, to allow multiple threads to make simultaneous progress (see “Processing Units and Threads” sidebar). Threads in a block can be spread across these processing units. Therefore, the hardware implementations of shared memory in these CUDA devices are typically designed to allow multiple processing units to simultaneously access its contents to support efficient data sharing among threads in a block. We will be learning several important types of parallel algorithms that can greatly benefit from such efficient data sharing among threads.

现在应该清楚了，寄存器、共享内存和全局内存都具有不同的功能、延迟和带宽。因此，了解如何声明变量以使其驻留在预期类型的​​内存中非常重要。表 5.1介绍了将程序变量声明到各种类型的设备内存中的 CUDA 语法。每个这样的声明还为其声明的 CUDA 变量提供了范围和生命周期。范围标识了可以访问该变量的线程范围：仅由单个线程，由块的所有线程，或由所有网格的所有线程。如果变量的作用域是单线程，则将为每个线程创建该变量的私有版本；每个线程只能访问该变量的私有版本。例如，如果内核声明了一个变量，其范围是一个线程，并且使用一百万个线程启动该变量，则将创建该变量的一百万个版本，以便每个线程初始化并使用自己的变量版本。

It should be clear by now that registers, shared memory, and global memory all have different functionalities, latencies, and bandwidth. It is, therefore, important to understand how to declare a variable so that it will reside in the intended type of memory. Table 5.1 presents the CUDA syntax for declaring program variables into the various types of device memory. Each such declaration also gives its declared CUDA variable a scope and lifetime. Scope identifies the range of threads that can access the variable: by a single thread only, by all threads of a block, or by all threads of all grids. If a variable’s scope is a single thread, a private version of the variable will be created for every thread; each thread can only access its private version of the variable. For example, if a kernel declares a variable of which the scope is a thread and it is launched with one million threads, one million versions of the variable will be created so that each thread initializes and uses its own version of the variable.

表 5.1 CUDA 变量类型限定符

Table 5.1 CUDA Variable Type Qualifiers

生命周期告诉变量可供使用时程序执行持续时间的一部分：无论是在内核执行中还是在整个应用程序中。如果变量的生命周期在内核执行期间，则必须在内核函数体内声明它，并且只能由内核代码使用。如果多次调用内核，则在这些调用中不会维护该变量的值。每次调用都必须初始化变量才能使用它们。另一方面，如果变量的生命周期贯穿整个应用程序，则必须在任何函数体之外声明它。这些变量的内容在应用程序的整个执行过程中得到维护，并且可供所有内核使用。

Lifetime tells the portion of the program’s execution duration when the variable is available for use: either within a kernel’s execution or throughout the entire application. If a variable’s lifetime is within a kernel’s execution, it must be declared within the kernel function body and will be available for use only by the kernel’s code. If the kernel is invoked several times, the value of the variable is not maintained across these invocations. Each invocation must initialize the variable to use them. On the other hand, if a variable’s lifetime is throughout the entire application, it must be declared outside of any function body. The contents of these variables are maintained throughout the execution of the application and available to all kernels.

如表 5.1所示，所有在内核和设备函数中声明的自动标量变量都被放入寄存器中。我们将非数组的变量称为标量变量。这些自动变量的范围在各个线程内。当内核函数声明自动变量时，将为每个线程生成该变量的私有副本执行内核函数。当线程终止时，它的所有自动变量也不再存在。在图 5.1中，变量Row、Col和Pvalue都是自动变量，属于这一类。请注意，访问这些变量的速度非常快并且是并行的，但必须小心不要超出硬件实现中寄存器存储的有限容量。我们将在第 6 章中讨论这一点。

As shown in Table 5.1, all automatic scalar variables declared in kernel and device functions are placed into registers. We refer to variables that are not arrays as scalar variables. The scopes of these automatic variables are within individual threads. When a kernel function declares an automatic variable, a private copy of that variable is generated for every thread that executes the kernel function. When a thread terminates, all its automatic variables also cease to exist. In Figure 5.1, variables Row, Col, and Pvalue are all automatic variables and fall into this category. Note that accessing these variables is extremely fast and parallel but one must be careful not to exceed the limited capacity of the register storage in the hardware implementations. We will address this point in Chapter 6.

自动数组变量不存储在寄存器中。²相反，它们存储在全局内存中，可能会导致较长的访问延迟和潜在的访问拥塞。与自动标量变量一样，这些数组的范围仅限于单个线程。也就是说，为每个线程创建并使用每个自动数组的私有版本。一旦线程终止执行，其自动数组变量的内容也不再存在。根据我们的经验，在内核函数和设备函数中很少需要使用自动数组变量。

Automatic array variables are not stored in registers.² Instead, they are stored into the global memory and may incur long access delays and potential access congestions. The scope of these arrays is, like automatic scalar variables, limited to individual threads. That is, a private version of each automatic array is created for and used by every thread. Once a thread terminates its execution, the contents of its automatic array variables also cease to exist. From our experience, one seldom needs to use automatic array variables in kernel functions and device functions.

如果变量声明前面有关键字 __shared__ （每个__由两个_字符组成），则它在 CUDA 中声明了一个共享变量。还可以在声明中的__shared__前面添加一个可选的__device__来达到相同的效果。这种声明通常驻留在内核函数或设备函数中。共享变量驻留在共享内存中。共享变量的作用域是在一个线程块内，也就是说，一个块中的所有线程都看到相同版本的共享变量。在内核执行期间，为每个线程块创建并使用共享变量的私有版本。共享变量的生命周期在内核的持续时间内。当内核终止执行时，其共享变量的内容将不再存在。正如我们之前讨论的，共享变量是块内线程相互协作的有效手段。从共享内存访问共享变量非常快并且高度并行。 CUDA 程序员经常使用共享变量来保存在内核执行阶段大量使用的全局内存数据部分。人们可能需要调整用于创建主要关注全局内存数据的一小部分的执行阶段的算法，正如我们将在第5.3 节中通过矩阵乘法进行演示的那样。

If a variable declaration is preceded by the keyword __shared__ (each __ consists of two _ characters), it declares a shared variable in CUDA. One can also add an optional __device__ in front of __shared__ in the declaration to achieve the same effect. Such declaration typically resides within a kernel function or a device function. Shared variables reside in shared memory. The scope of a shared variable is within a thread block, that is, all threads in a block see the same version of a shared variable. A private version of the shared variable is created for and used by each thread block during kernel execution. The lifetime of a shared variable is within the duration of the kernel. When a kernel terminates its execution, the contents of its shared variables cease to exist. As we discussed earlier, shared variables are an efficient means for threads within a block to collaborate with each other. Accessing shared variables from the shared memory is extremely fast and highly parallel. CUDA programmers often use shared variables to hold the portion of global memory data that are heavily used in an execution phase of a kernel. One may need to adjust the algorithms used to create execution phases that heavily focus on small portions of the global memory data, as we will demonstrate with matrix multiplication in Section 5.3.

如果变量声明前面有关键字__constant__（每个__由两个_字符组成），则它在 CUDA 中声明一个常量变量。还可以在__constant__前面添加一个可选的__device__来达到相同的效果。常量变量的声明必须是在任何函数体之外。常量变量的作用域是所有网格，这意味着所有网格中的所有线程都看到相同版本的常量变量。常量变量的生命周期是整个应用程序执行的时间。常量变量通常用于为核函数提供输入值的变量。常量变量存储在全局内存中，但会被缓存以便有效访问。通过适当的访问模式，访问常量内存的速度非常快并且是并行的。目前，应用程序中常量变量的总大小限制为 65,536 字节。我们可能需要分解输入数据量以适应这一限制，正如我们将在第 8 章中说明的那样。

If a variable declaration is preceded by the keyword __constant__ (each __ consists of two _ characters), it declares a constant variable in CUDA. One can also add an optional __device__ in front of __constant__ to achieve the same effect. Declaration of constant variables must be outside any function body. The scope of a constant variable is all grids, meaning that all threads in all grids see the same version of a constant variable. The lifetime of a constant variable is the entire application execution. Constant variables are often used for variables that provide input values to kernel functions. Constant variables are stored in the global memory but are cached for efficient access. With appropriate access patterns, accessing constant memory is extremely fast and parallel. Currently, the total size of constant variables in an application is limited at 65,536 bytes. One may need to break up the input data volume to fit within this limitation, as we will illustrate in Chapter 8.

声明前面仅带有关键字__device__（每个__由两个_字符组成）的变量是全局变量，将被放置在全局内存中。对全局变量的访问速度很慢。在较新的设备中，通过缓存改进了访问全局变量的延迟和吞吐量。全局变量的一个重要优点是它们对所有内核的所有线程都是可见的。它们的内容也会在整个执行过程中持续存在。因此，全局变量可以用作线程跨块协作的手段。然而，我们必须意识到这样一个事实：除了终止当前内核执行之外，目前还没有一种简单的方法可以在不同线程块的线程之间进行同步，或者在访问全局内存时确保线程之间的数据一致性。³因此，全局变量通常用于将信息从一个内核调用传递到另一内核调用。

A variable of which the declaration is preceded only by the keyword __device__ (each __ consists of two _ characters) is a global variable and will be placed in the global memory. Accesses to a global variable are slow. Latency and throughput of accessing global variables have been improved with caches in more recent devices. One important advantage of global variables is that they are visible to all threads of all kernels. Their contents also persist through the entire execution. Thus, global variables can be used as a means for threads to collaborate across blocks. One must, however, be aware of the fact that there is currently no easy way to synchronize between threads from different thread blocks or to ensure data consistency across threads when accessing global memory other than terminating the current kernel execution.³ Therefore, global variables are often used to pass information from one kernel invocation to another kernel invocation.

在CUDA中，指针用于指向全局内存中的数据对象。在内核和设备函数中，指针的使用有两种典型的方式。首先，如果一个对象是由主机函数分配的，则指向该对象的指针由cudaMalloc()初始化，并且可以作为参数传递给内核函数。例如，图 5.1中的参数d_M、d_N和d_P就是这样的指针。第二种用法是将全局内存中声明的变量的地址分配给指针变量。例如，内核函数中的语句{float* ptr = &GlobalVar;}将GlobalVar的地址分配给自动指针变量ptr。有关在其他内存类型中使用指针的信息，读者应参阅CUDA 编程指南。

In CUDA, pointers are used to point to data objects in global memory. There are two typical ways in which pointer usage arises in kernel and device functions. First, if an object is allocated by a host function, the pointer to the object is initialized by cudaMalloc() and can be passed to the kernel function as a parameter. For example, the parameters d_M, d_N, and d_P in Figure 5.1 are such pointers. The second type of usage is to assign the address of a variable declared in the global memory to a pointer variable. For example, the statement {float∗ ptr = &GlobalVar;} in a kernel function assigns the address of GlobalVar into an automatic pointer variable ptr. Readers should refer to the CUDA Programming Guide for using pointers in other memory types.

5.3 减少全局内存流量的策略

5.3 A Strategy for Reducing Global Memory Traffic

在 CUDA 中使用设备内存时，我们有一个内在的权衡：全局内存大但速度慢，而共享内存小但速度快。一种常见的策略是将数据划分为称为图块的子集，以便每个图块适合共享内存。术语瓦片借鉴了这样的类比：一堵大墙（即，全局存储器数据）可以被瓦片（即，每个都可以放入共享存储器的子集）覆盖。一个重要的标准是这些图块上的内核计算可以彼此独立地完成。请注意，在给定任意核函数的情况下，并非所有数据结构都可以划分为图块。

We have an intrinsic trade-off in the use of device memories in CUDA: global memory is large but slow, whereas the shared memory is small but fast. A common strategy is partition the data into subsets called tiles so that each tile fits into the shared memory. The term tile draws on the analogy that a large wall (i.e., the global memory data) can be covered by tiles (i.e., subsets that each can fit into the shared memory). An important criterion is that the kernel computation on these tiles can be done independently of each other. Note that not all data structures can be partitioned into tiles given an arbitrary kernel function.

平铺的概念可以用矩阵乘法的例子来说明。图 5.5显示了矩阵乘法的一个小例子。对应的核函数如图5.1所示。为简洁起见，我们将d_P[y*Width+x]、d_M[y*Width+x ]和d_N[y*Width+x]分别缩写为 P _y,x、M _y,x和 N _y,x。此示例假设我们使用四个 2×2 块来计算 P 矩阵。图 5.5突出显示了 block(0,0) 的四个线程完成的计算。这四个线程计算 P _0,0、 P _0,1、 P _1,0和 P _1,1。block(0,0) 的 thread(0,0) 和 thread(0,1) 对M和N元素的访问用黑色箭头突出显示。例如， thread(0,0) 读取 M _0,0和 N _0,0，然后是 M _0,1和 N _1,0，然后是 M _0,2和 N _2,0，然后是 M _0,3和N _3,0。

The concept of tiling can be illustrated with the matrix multiplication example. Figure 5.5 shows a small example of matrix multiplication. It corresponds to the kernel function in Figure 5.1. For brevity, we abbreviate d_P[y∗Width+x], d_M[y∗Width+x], and d_N[y∗Width+x] into P_y,x, M_y,x, and N_y,x, respectively. This example assumes that we use four 2×2 blocks to compute the P matrix. Figure 5.5 highlights the computation done by the four threads of block(0,0). These four threads compute P_0,0, P_0,1, P_1,0, and P_1,1. The accesses to the M and N elements by thread(0,0) and thread(0,1) of block(0,0) are highlighted with black arrows. For example, thread(0,0) reads M_0,0 and N_0,0, followed by M_0,1, and N_1,0 followed by M_0,2 and N_2,0, followed by M_0,3 and N_3,0.

图 5.6显示了块_0,0中所有线程完成的全局内存访问。线程按垂直方向列出，访问时间在水平方向向右增加。请注意，每个线程在执行期间都会访问M的四个元素和N的四个元素。在突出显示的四个线程中，它们访问的M和N元素存在显着重叠。例如，线程_0,0和线程_0,1都访问 M _0,0以及M的第 0 行的其余部分。同样，线程_0,1和线程_1,1都访问 N _0,1以及N的第1列的其余部分。

Figure 5.6 shows the global memory accesses done by all threads in block_0,0. The threads are listed in the vertical direction, with time of access increasing to the right in the horizontal direction. Note that each thread accesses four elements of M and four elements of N during its execution. Among the four threads highlighted, there is a significant overlap in terms of the M and N elements they access. For example, thread_0,0 and thread_0,1 both access M_0,0 as well as the rest of row 0 of M. Similarly, thread_0,1 and thread_1,1 both access N_0,1 as well as the rest of column 1 of N.

图 5.1中的内核被编写为线程_0,0和线程_0,1都从全局内存访问M的第 0 行元素。如果我们能够以某种方式设法让线程_0,0和线程_1,0协作，以便这些M 个元素仅从全局内存加载一次，那么我们可以将全局内存的访问总数减少一半。一般来说，我们可以看到，在块_0,0的执行过程中，每个M和N元素都被访问了两次。因此，如果我们可以让所有四个线程协作访问全局内存，我们就可以将全局内存的流量减少一半。

The kernel in Figure 5.1 is written so that both thread_0,0 and thread_0,1 access row 0 elements of M from the global memory. If we can somehow manage to have thread_0,0 and thread_1,0 to collaborate so that these M elements are only loaded from global memory once, we can reduce the total number of accesses to the global memory by half. In general, we can see that every M and N element is accessed exactly twice during the execution of block_0,0. Therefore, if we can have all four threads to collaborate in their accesses to global memory, we can reduce the traffic to the global memory by half.

读者应该验证矩阵乘法示例中全局内存流量的潜在减少与所使用的块的尺寸成正比。对于N × N块，全局内存流量的潜在减少将为N。也就是说，如果我们使用 16×16 块，则可以通过线程之间的协作将全局内存流量减少到 1/16。

Readers should verify that the potential reduction in global memory traffic in the matrix multiplication example is proportional to the dimension of the blocks used. With N×N blocks, the potential reduction of global memory traffic would be N. That is, if we use 16×16 blocks, one can potentially reduce the global memory traffic to 1/16 through collaboration between threads.

交通拥堵显然不仅仅出现在计算领域。我们大多数人都经历过高速公路系统的交通拥堵，如图5.7所示。高速公路交通拥堵的根本原因是，太多的汽车挤在一条为车辆数量少得多的道路设计的道路上。当发生拥堵时，出行时间每辆车都大大增加。在交通拥堵期间，上班时间很容易增加一倍或三倍。

Traffic congestion obviously does not only arise in computing. Most of us have experienced traffic congestion in highway systems, as illustrated in Figure 5.7. The root cause of highway traffic congestion is that there are too many cars all squeezing through a road that is designed for a much smaller number of vehicles. When congestion occurs, the travel time for each vehicle is greatly increased. Commute time to work can easily double or triple during traffic congestion.

所有提出的减少交通拥堵的解决方案都涉及减少道路上的汽车。假设通勤人数不变，人们需要拼车以减少道路上的汽车数量。在美国，一种常见的拼车方式是拼车，一群通勤者轮流驾驶一辆车去上班。在一些国家，政府只是禁止某些类别的汽车每天上路。例如，星期一、星期三或星期五可能不允许牌照奇数的汽车上路。这鼓励了在不同日期允许使用汽车的人组成拼车团体。也有一些国家的政府让汽油变得非常昂贵，以至于人们为了省钱而拼车。在其他国家，政府可能会对减少道路上汽车数量的行为提供激励。在美国，拥堵高速公路的一些车道被指定为拼车车道，只有两到三人以上的汽车才允许使用这些车道。所有这些鼓励拼车的措施都是为了克服拼车需要额外努力的事实，如图5.8所示。

All proposed solutions for reduced traffic congestion involve reduction of cars on the road. Assuming that the number of commuters is constant, people need to share rides to reduce the number of cars on the road. A common way to share rides in the United States is carpools, where a group of commuters take turns to drive the group to work in one vehicle. In some countries, the government simply disallows certain classes of cars to be on the road on a daily basis. For example, cars with odd license plates may not be allowed on the road on Monday, Wednesday, or Friday. This encourages people whose cars are allowed on different days to form a carpool group. There are also countries where the government makes gasoline so expensive that people form carpools to save money. In other countries, the government may provide incentives for behavior that reduces the number of cars on the road. In the United States, some lanes of congested highways are designated as carpool lanes—only cars with more than two or three people are allowed to use these lanes. All these measures for encouraging carpooling are designed to overcome the fact that carpooling requires extra effort, as we show in Figure 5.8.

图 5.8的上半部分显示了一个良好的拼车时间表模式。时间从左向右。工人A和工人B有类似的情况睡眠、工作和晚餐的时间表。这使得这两名工人可以轻松地用一辆车上下班和回家。他们相似的时间表使他们更容易就共同的出发时间和返回时间达成一致。然而，图 5.8下半部分所示的时间表并非如此。在这种情况下，工人 A 和工人 B 的习惯截然不同。工人A聚会到天亮，白天睡觉，晚上上班。工人 B 晚上睡觉，早上上班，下午 6 点回家吃晚饭。时间安排相差太大，两个工人根本无法协调同一时间开一辆车上班和回家。 。为了让这些工人组成拼车，他们需要协商一个类似于图 5.8上半部分所示的共同时间表。

The top half of Figure 5.8 shows a good schedule pattern for carpooling. Time goes from left to right. Worker A and worker B have similar schedules for sleep, work, and dinner. This allows these two workers to easily go to work and return home in one car. Their similar schedules allow them to more easily agree on a common departure time and return time. This is, however, not the case of the schedules shown in the bottom half of Figure 5.8. Worker A and worker B have very different habits in this case. Worker A parties until sunrise, sleeps during the day, and goes to work in the evening. Worker B sleeps at night, goes to work in the morning, and returns home for dinner at 6 p.m. The schedules are so wildly different that there is no way these two workers can coordinate a common time to drive to work and return home in one car. For these workers to form a carpool, they need to negotiate a common schedule similar to what is shown in the top half of Figure 5.8.

平铺算法与拼车安排非常相似。我们可以将每个线程访问的数据值视为通勤者，将请求的 DRAM 视为车辆。当DRAM请求的速率超过DRAM系统的规定带宽时，就会出现流量拥塞并且运算单元变得空闲。如果多个线程从同一 DRAM 位置访问数据，它们可以形成“拼车”并将其访问合并到一个 DRAM 请求中。然而，这要求线程具有相似的执行调度，以便它们的数据访问可以合并为一个。如图 5.9所示，其中顶部显示两个线程以相似的时序访问相同的数据元素。下半部分显示了两个线程在不同的时间访问其公共数据。下半部分排列不好的原因是从 DRAM 带回的数据元素需要长时间保存在片上内存中，等待线程 2 消耗它们。这可能需要保留大量数据元素，因此需要大量的片上存储器。正如我们将在下一节中展示的，我们将使用屏障同步来保持形成“拼车”组的线程遵循大致相同的执行时序。

Tiled algorithms are very similar to carpooling arrangements. We can think of data values accessed by each thread as commuters and DRAM requested as vehicles. When the rate of DRAM requests exceeds the provisioned bandwidth of the DRAM system, traffic congestion arises and the arithmetic units become idle. If multiple threads access data from the same DRAM location, they can form a “carpool” and combine their accesses into one DRAM request. This, however, requires the threads to have a similar execution schedule so that their data accesses can be combined into one. This is shown in Figure 5.9, where the top portion shows two threads that access the same data elements with similar timing. The bottom half shows two threads that access their common data in very different times. The reason why the bottom half is a bad arrangement is that data elements brought back from the DRAM need to be kept in the on-chip memory for a long time, waiting for thread 2 to consume them. This will likely require a large number of data elements to be kept around, thus large on-chip memory requirements. As we will show in the next section, we will use barrier synchronization to keep the threads that form the “carpool” group to follow approximately the same execution timing.

5.4 平铺矩阵——矩阵乘法内核

5.4 A Tiled Matrix–Matrix Multiplication Kernel

我们现在提出一种算法，其中线程协作以减少全局内存的流量。基本思想是让线程协作将M和N 个元素加载到共享内存中，然后再在点积计算中单独使用这些元素。请记住，共享内存的大小非常小，在将这M和N个元素加载到共享内存中时必须小心，不要超出共享内存的容量。这可以通过将 M 和 N 矩阵划分为更小的块来实现。这些图块的大小经过选择，以便它们可以适合共享内存。在最简单的形式中，图块尺寸等于块的尺寸，如图5.10所示。

We now present an algorithm where threads collaborate to reduce the traffic to the global memory. The basic idea is to have the threads to collaboratively load M and N elements into the shared memory before they individually use these elements in their dot product calculation. Keep in mind that the size of the shared memory is quite small and one must be careful not to exceed the capacity of the shared memory when loading these M and N elements into the shared memory. This can be accomplished by dividing the M and N matrices into smaller tiles. The size of these tiles is chosen so that they can fit into the shared memory. In the simplest form, the tile dimensions equal those of the block, as illustrated in Figure 5.10.

在图 5.10中，我们将 M 和 N 矩阵划分为 2×2 块，如粗线所示。点积计算由每个线程现在分为几个阶段。在每个阶段，块中的所有线程协作将M 个元素的图块和N 个元素的图块加载到共享内存中。这是通过让块中的每个线程将一个M元素和一个N元素加载到共享内存中来完成的，如图5.11所示。图5.11的每一行显示了执行过程线程的活动。请注意，时间是从左向右进行的。我们只需要显示块_0,0中线程的活动；其他块都有相同的行为。M 个元素的共享内存数组称为Mds。N 个元素的共享内存数组称为Nds。在第 1 阶段开始时，块_{0,0的四个线程协作将}M 个元素的图块加载到共享内存中：线程_0,0将 M _0,0加载到 Mds _0,0中，线程_0,1加载 M _0,1到 Mds _0,1中，线程_1,0将 M _1,0加载到 Mds _1,0中，线程_1,1将 M _1,1加载到 Mds _1,1中。见图5.11第二列。N 个元素的图块也以类似的方式加载，如图 5.11的第三列所示。

In Figure 5.10, we divide the M and N matrices into 2×2 tiles, as delineated by the thick lines. The dot product calculations performed by each thread are now divided into phases. In each phase, all threads in a block collaborate to load a tile of M elements and a tile of N elements into the shared memory. This is done by having every thread in a block to load one M element and one N element into the shared memory, as illustrated in Figure 5.11. Each row of Figure 5.11 shows the execution activities of a thread. Note that time progresses from left to right. We only need to show the activities of threads in block_0,0; the other blocks all have the same behavior. The shared memory array for the M elements is called Mds. The shared memory array for the N elements is called Nds. At the beginning of phase 1, the four threads of block_0,0 collaboratively load a tile of M elements into shared memory: thread_0,0 loads M_0,0 into Mds_0,0, thread_0,1 loads M_0,1 into Mds_0,1, thread_1,0 loads M_1,0 into Mds_1,0, and thread_1,1 loads M_1,1 into Mds_1,1. See the second column of Figure 5.11. A tile of N elements is also loaded in a similar manner, shown in the third column of Figure 5.11.

将M和N元素的两个图块加载到共享内存后，这些值将用于计算点积。请注意，共享内存中的每个值都会使用两次。例如，由线程_1,1加载到 Mds _1,1中的M _1,1值使用两次，一次由线程_0,1使用，一次由线程_1,1使用。通过将每个全局内存值加载到共享内存中以便可以多次使用，我们减少了对全局内存的访问次数。在这种情况下，我们将对全局内存的访问次数减少一半。如果图块是N × N元素，读者应验证是否减少了N倍。

After the two tiles of M and N elements are loaded into the shared memory, these values are used in the calculation of the dot product. Note that each value in the shared memory is used twice. For example, the M_1,1 value, loaded by thread_1,1 into Mds_1,1, is used twice, once by thread_0,1 and once by thread_1,1. By loading each global memory value into shared memory so that it can be used multiple times, we reduce the number of accesses to the global memory. In this case, we reduce the number of accesses to the global memory by half. Readers should verify that the reduction is by a factor of N if the tiles are N×N elements.

请注意，图 5.6中每个点积的计算现在分两个阶段执行，如图5.11中的阶段 1 和阶段 2 所示。在每个阶段，两对输入矩阵元素的乘积被累加到Pvalue变量中。请注意，Pvalue是一个自动变量，因此会为每个线程生成一个私有版本。我们添加下标只是为了澄清这些是为每个线程创建的Pvalue变量的不同实例。第一阶段计算如图5.11第四栏所示；第二阶段在第七列。一般来说，如果输入矩阵的维度为N并且图块大小为TILE_WIDTH，则点积将在N/TILE_WIDTH阶段中执行。这些阶段的创建是减少对全局存储器的访问的关键。每个阶段都专注于输入矩阵值的一个小子集，线程可以协作地将子集加载到共享内存中，并使用共享内存中的值来满足该阶段中的重叠输入需求。

Note that the calculation of each dot product in Figure 5.6 is now performed in two phases, shown as phase 1 and phase 2 in Figure 5.11. In each phase, products of two pairs of the input matrix elements are accumulated into the Pvalue variable. Note that Pvalue is an automatic variable so a private version is generated for each thread. We added subscripts just to clarify that these are different instances of the Pvalue variable created for each thread. The first phase calculation is shown in the fourth column of Figure 5.11; the second phase in the seventh column. In general, if an input matrix is of dimension N and the tile size is TILE_WIDTH, the dot product would be performed in N/TILE_WIDTH phases. The creation of these phases is key to the reduction of accesses to the global memory. With each phase focusing on a small subset of the input matrix values, the threads can collaboratively load the subset into the shared memory and use the values in the shared memory to satisfy their overlapping input needs in the phase.

另请注意，Mds和Nds被重新用于保存输入值。在每个阶段中，相同的Mds和Nds用于保存该阶段中使用的M和N元素的子集。这允许更小的共享内存来服务对全局内存的大部分访问。这是因为每个阶段都专注于输入矩阵元素的一小部分。这种集中的访问行为称为局部性。当算法表现出局部性，有机会使用小型高速存储器来服务大多数访问并将这些访问从全局存储器中删除。局部性对于在多核 CPU 中实现高性能与在多线程 GPU 中一样重要。我们回到第 6 章中的局部性概念。

Note also that Mds and Nds are reused to hold the input values. In each phase, the same Mds and Nds are used to hold the subset of M and N elements used in the phase. This allows a much smaller shared memory to serve most of the accesses to global memory. This is due to the fact that each phase focuses on a small subset of the input matrix elements. Such focused access behavior is called locality. When an algorithm exhibits locality, there is an opportunity to use small, high-speed memories to serve most of the accesses and remove these accesses from the global memory. Locality is as important for achieving high performance in multicore CPUs as in many-thread GPUs. We return to the concept of locality in Chapter 6.

我们现在准备展示使用共享内存来减少全局内存流量的平铺内核函数。图 5.12所示的内核实现了图 5.11所示的阶段。在图 5.12中，第 1 行和第 2 行将Mds和Nds声明为共享内存变量。回想一下，共享内存变量的范围是一个块。因此，将为每个块创建一对Mds和Nds，并且块的所有线程都可以访问相同的Mds和Nds。这很重要，因为块中的所有线程都必须有权访问由其对等方加载到Mds和Nds中的M和N值，以便它们可以使用这些值来满足其输入需求。

We are now ready to present the tiled kernel function that uses shared memory to reduce the traffic to global memory. The kernel shown in Figure 5.12 implements the phases illustrated in Figure 5.11. In Figure 5.12, lines 1 and 2 declare Mds and Nds as shared memory variables. Recall that the scope of shared memory variables is a block. Thus, one pair of Mds and Nds will be created for each block and all threads of a block have access to the same Mds and Nds. This is important since all threads in a block must have access to the M and N values loaded into Mds and Nds by their peers so that they can use these values to satisfy their input needs.

#定义TILE_WIDTH 16

#define TILE_WIDTH 16

第 3 行和第 4 行将threadIdx和blockIdx值保存到自动变量中，从而保存到寄存器中以便快速访问。回想一下自动标量变量被放入寄存器中。它们的范围是在每个单独的线程中。也就是说，运行时系统为每个线程创建一个私有版本的tx、ty、bx和by 。它们将驻留在可由一个线程访问的寄存器中。它们使用threadIdx和blockIdx值进行初始化，并在线程的生命周期内多次使用。一旦线程结束，这些变量的值也不再存在。

Lines 3 and 4 save the threadIdx and blockIdx values into automatic variables and thus into registers for fast access. Recall that automatic scalar variables are placed into registers. Their scope is in each individual thread. That is, one private version of tx, ty, bx, and by is created by the runtime system for each thread. They will reside in registers that are accessible by one thread. They are initialized with the threadIdx and blockIdx values and used many times during the lifetime of the thread. Once the thread ends, the values of these variables also cease to exist.

第 5 行和第 6 行确定线程要生成的d_P元素的行索引和列索引。如第 6 行所示，线程要生成的d_P元素的水平 ( x ) 位置或列索引可以计算为bx*TILE_WIDTH+tx。这是因为每个块在水平维度上覆盖TILE_WIDTH元素。 bx块中的线程在其之前将有bx 个线程块，或(bx*TILE_WIDTH)个线程；它们覆盖d_P的bx*TILE_WIDTH元素。同一块内的另一个tx线程将覆盖d_P的另一个tx元素。因此，具有bx和tx的线程应该负责计算x索引为bx*TILE_WIDTH+tx的d_P元素。该水平索引保存在线程的变量Col （列）中，如图5.13所示。对于图5.10中的示例，块_{1,0的线程0,1}要计算的d_P元素的x索引是0×2+1=1。类似地，y索引可以通过*TILE_WIDTH+ty计算。该垂直索引保存在线程的变量Row中。因此，如图5.10所示，每个线程计算Col列和Row行处的d_P元素。回到图5.10中的示例，块_0,1的线程_1,0要计算的d_P元素的y索引是1×2+0=2。因此，该线程要计算的d_P元素是d_P _2,1。

Lines 5 and 6 determine the row index and column index of the d_P element that the thread is to produce. As shown in line 6, the horizontal (x) position, or the column index of the d_P element to be produced by a thread, can be calculated as bx∗TILE_WIDTH+tx. This is because each block covers TILE_WIDTH elements in the horizontal dimension. A thread in block bx would have bx blocks of threads, or (bx∗TILE_WIDTH) threads, before it; they cover bx∗TILE_WIDTH elements of d_P. Another tx thread within the same block would cover another tx element of d_P. Thus, the thread with bx and tx should be responsible for calculating the d_P element of which the xindex is bx∗TILE_WIDTH+tx. This horizontal index is saved in the variable Col (for column) for the thread and is also illustrated in Figure 5.13. For the example in Figure 5.10, the x index of the d_P element to be calculated by thread_0,1 of block_1,0 is 0×2+1=1. Similarly, the y index can be calculated as by∗TILE_WIDTH+ty. This vertical index is saved in the variable Row for the thread. Thus, as shown in Figure 5.10, each thread calculates the d_P element at the Col column and the Row row. Going back to the example in Figure 5.10, the y index of the d_P element to be calculated by thread_1,0 of block_0,1 is 1×2+0=2. Thus, the d_P element to be calculated by this thread is d_P_2,1.

图 5.12的第 8行标记了循环的开始，该循环迭代计算最终d_P元素的所有阶段。循环的每次迭代对应于图 5.11所示的计算的一个阶段。m变量表示点积已完成的阶段数。回想一下，每个阶段都使用d_M 元素的一个图块和d_N元素的一个图块。因此，在每个阶段开始时，前面的阶段已处理了m*TILE_WIDTH对d_M和d_N元素。

Line 8 of Figure 5.12 marks the beginning of the loop that iterates through all the phases of calculating the final d_P element. Each iteration of the loop corresponds to one phase of the calculation shown in Figure 5.11. The m variable indicates the number of phases that have already been done for the dot product. Recall that each phase uses one tile of d_M and one tile of d_N elements. Therefore, at the beginning of each phase, m∗TILE_WIDTH pairs of d_M and d_N elements have been processed by previous phases.

在每个阶段，第 9 行将适当的d_M元素加载到共享内存中。由于我们已经知道线程要处理的d_M的行和d_N的列，因此我们将重点关注d_M的列索引和d_N的行索引。如图5.11所示，每个块都有TILE_WIDTH ^{2 个}线程，它们将协作将TILE_WIDTH ² d_M元素加载到共享内存中。因此，我们需要做的就是分配每个线程加载一个d_M元素。使用blockIdx和线程Idx。请注意，要加载的d_M元素部分的起始列索引是m*TILE_WIDTH。因此，一种简单的方法是让每个线程从包含threadIdx.x值的偏移量tx处加载一个元素。这正是我们在第 9 行中所看到的，其中每个线程加载d_M[Row*Width + m*TILE_WIDTH + tx]。由于Row的值是ty的线性函数，因此每个TILE_WIDTH ²线程都会将唯一的d_M元素加载到共享内存中。这些线程将一起加载图 5.13中d_M的暗方形子集。读者应该使用图 5.5和5.6中的小示例来验证地址计算是否正确。

In each phase, line 9 loads the appropriate d_M element into the shared memory. Since we already know the row of d_M and column of d_N to be processed by the thread, we will focus on the column index of d_M and row index of d_N. As shown in Figure 5.11, each block has TILE_WIDTH² threads that will collaborate to load TILE_WIDTH² d_M elements into the shared memory. Thus, all we need to do is to assign each thread to load one d_M element. This is conveniently done using the blockIdx and threadIdx. Note that the beginning column index of the section of d_M elements to be loaded is m∗TILE_WIDTH. Therefore, an easy approach is to have every thread load an element from at an offset tx that contains threadIdx.x value. This is precisely what we have in line 9, where each thread loads d_M[Row∗Width + m∗TILE_WIDTH + tx]. Since the value of Row is a linear function of ty, each of the TILE_WIDTH² threads will load a unique d_M element into the shared memory. Together, these threads will load the dark square subset of d_M in Figure 5.13. Readers should use the small example in Figures 5.5 and 5.6 to verify that the address calculation works correctly.

第 11 行中的屏障__syncthreads()确保所有线程都已完成将d_M和d_N的图块加载到Mds和Nds中，然后才能继续前进。然后，第 12 行中的循环基于这些图块元素执行点积的一阶段。thread(ty,tx)的循环过程如图5.13所示，其中d_M和d_N元素沿标有k （第 12 行中的循环变量）的箭头使用。请注意，这些元素将从Mds和Nds （保存这些d_M和d_N元素的共享内存数组）访问。第 14 行中的屏障__syncthreads()确保所有线程都已完成对共享内存中d_M和d_N元素的使用，然后再进行下一次迭代并加载下一个图块中的元素。这样，任何线程都不会过早加载元素并损坏其他线程的输入值。

The barrier __syncthreads() in line 11 ensures that all threads have finished loading the tiles of d_M and d_N into Mds and Nds before any of them can move forward. The loop in line 12 then performs one phase of the dot product based on these tile elements. The progression of the loop for thread(ty,tx) is shown in Figure 5.13, with the direction of d_M and d_N elements usage along the arrow marked with k, the loop variable in line 12. Note that these elements will be accessed from Mds and Nds, the shared memory arrays holding these d_M and d_N elements. The barrier __syncthreads() in line 14 ensures that all threads have finished using the d_M and d_N elements in the shared memory before any of them move on to the next iteration and load the elements in the next tiles. This way, none of the threads would load the elements too early and corrupt the input values for other threads.

点积的所有部分完成后，执行退出第 8 行的循环。所有线程使用Row和Col写入其d_P元素。

After all sections of the dot product are complete, the execution exits the loop of line 8. All threads write to their d_P element using the Row and Col.

平铺算法的好处是巨大的。对于矩阵乘法，全局内存访问减少了TILE_WIDTH倍。如果使用 16×16 块，我们可以将全局内存访问减少 16 倍。这将 CGMA 从 1 增加到 16。这一改进允许 CUDA 设备的内存带宽支持接近其峰值性能的计算速率。例如，这一改进使得 150 GB/s 的全局内存带宽能够支持 (150/4)×16=600 GFLOPS！

The benefit of the tiled algorithm is substantial. For matrix multiplication, the global memory accesses are reduced by a factor of TILE_WIDTH. If one uses 16×16 tiles, we can reduce the global memory accesses by a factor of 16. This increases the CGMA from 1 to 16. This improvement allows the memory bandwidth of a CUDA device to support a computation rate close to its peak performance. For example, this improvement allows a 150 GB/s global memory bandwidth to support (150/4)×16=600 GFLOPS!

5.5 内存作为并行性的限制因素

5.5 Memory as a Limiting Factor to Parallelism

虽然 CUDA 寄存器和共享内存可以非常有效地减少对全局内存的访问次数，但必须小心不要超出这些内存的容量。这些内存是线程执行所需的资源形式。每个 CUDA 设备提供的资源数量有限，这限制了给定应用程序可以同时驻留在 SM 中的线程数量。一般来说，每个线程需要的资源越多，每个SM中可以驻留的线程数量就越少，因此整个设备中可以驻留的线程数量就越少。

While CUDA registers and shared memory can be extremely effective in reducing the number of accesses to global memory, one must be careful not to exceed the capacity of these memories. These memories are forms of resources that are needed for thread execution. Each CUDA device offers a limited amount of resources, which limits the number threads that can simultaneously reside in the SM for a given application. In general, the more resources each thread requires, the fewer the number of threads can reside in each SM, and thus the fewer number of threads that can reside in the entire device.

让我们用一个例子来说明内核的寄存器使用和设备可以支持的并行级别之间的交互。假设在设备D中，每个SM最多可以容纳1536个线程，并且有16384个寄存器。虽然 16,384 是一个很大的数字，但考虑到每个 SM 中可以驻留的线程数量，它只允许每个线程使用非常有限数量的寄存器。为了支持 1,536 个线程，每个线程只能使用 16,384÷1,536=10 个寄存器。如果每个线程使用11个寄存器，那么每个SM中能够并发执行的线程数量将会减少。这种减少是在块粒度上完成的。例如，如果每个块包含512个线程，线程的减少将通过一次减少512个线程来完成。因此，1,536 个线程的下一个数量将是 512 个，即可以同时驻留在每个 SM 中的线程数量减少了三分之一。这可以大大减少可用于调度的扭曲数量，从而降低处理器在存在长延迟操作时找到有用工作的能力。

Let’s use an example to illustrate the interaction between register usage of a kernel and the level of parallelism that a device can support. Assume that in a device D, each SM can accommodate up to 1,536 threads and has 16,384 registers. While 16,384 is a large number, it only allows each thread to use a very limited number of registers considering the number of threads that can reside in each SM. To support 1,536 threads, each thread can use only 16,384÷1,536=10 registers. If each thread uses 11 registers, the number of threads able to be executed concurrently in each SM will be reduced. Such reduction is done at the block granularity. For example, if each block contains 512 threads, the reduction of threads will be done by reducing 512 threads at a time. Thus, the next lower number of threads from 1,536 would be 512, a one-third reduction of threads that can simultaneously reside in each SM. This can greatly reduce the number of warps available for scheduling, thus reducing the processor’s ability to find useful work in the presence of long-latency operations.

请注意，每个 SM 可用的寄存器数量因设备而异。应用程序可以动态确定所用设备的每个 SM 中可用的寄存器数量，并选择使用适合该设备的寄存器数量的内核版本。这可以通过调用cudaGetDeviceProperties()函数来完成，该函数的使用已在第 4.6 节中讨论。假设变量&dev_prop被传递给设备属性的函数，并且字段dev_prop.regsPerBlock给出了每个 SM 中可用的寄存器数量。对于设备 D，该字段的返回值应为 16,384。然后，应用程序可以将该数量除以驻留在每个 SM 中的目标线程数量，以确定可以在内核中使用的寄存器数量。

Note that the number of registers available to each SM varies from device to device. An application can dynamically determine the number of registers available in each SM of the device used and choose a version of the kernel that uses the number of registers appropriate for the device. This can be done by calling the cudaGetDeviceProperties() function, the use of which was discussed in Section 4.6. Assume that variable &dev_prop is passed to the function for the device property, and the field dev_prop.regsPerBlock gives the number of registers available in each SM. For device D, the returned value for this field should be 16,384. The application can then divide this number by the target number of threads to reside in each SM to determine the number of registers that can be used in the kernel.

共享内存的使用还可以限制分配给每个 SM 的线程数量。假设设备 D 在每个 SM 中具有 16,384 (16 K) 字节的共享内存。请记住，共享内存是按块使用的。假设每个SM最多可容纳8个块。要达到此最大值，每个块不得使用超过 2 K 字节的共享内存。如果每个块使用超过2K字节的内存，则每个SM中可以驻留的块的数量使得这些块使用的共享内存总量不超过16K字节。例如，如果每个块使用 5 K 字节的共享内存，则不能为每个 SM 分配超过 3 个块。

Shared memory usage can also limit the number of threads assigned to each SM. Assume device D has 16,384 (16 K) bytes of shared memory in each SM. Keep in mind that shared memory is used by blocks. Assume that each SM can accommodate up to eight blocks. To reach this maximum, each block must not use more than 2 K bytes of shared memory. If each block uses more than 2 K bytes of memory, the number of blocks that can reside in each SM is such that the total amount of shared memory used by these blocks does not exceed 16 K bytes. For example, if each block uses 5 K bytes of shared memory, no more than three blocks can be assigned to each SM.

对于矩阵乘法示例，共享内存可能成为限制因素。对于 16×16 的图块大小，每个块需要 16×16×4=1 K 字节的存储空间来存储Mds。Nds还需要 1 KB 。因此，每个块使用 2 K 字节的共享内存。 16 K 字节共享内存允许八个块同时驻留在 SM 中。由于这与线程硬件允许的最大值相同，因此共享内存不是该图块大小的限制因素。在这种情况下，真正的限制是线程硬件限制，即每个 SM 中仅允许 768 个线程。这将每个 SM 中的块数量限制为三个。因此，仅使用 3×2 KB=6 KB 的共享内存。这些限制确实会随着下一代设备的变化而变化，但可以在运行时确定的属性，例如GT200系列处理器在每个SM中最多可以支持1,024个线程。

For the matrix multiplication example, shared memory can become a limiting factor. For a tile size of 16×16, each block needs a 16×16×4=1 K bytes of storage for Mds. Another 1 KB is needed for Nds. Thus, each block uses 2 K bytes of shared memory. The 16 K–byte shared memory allows eight blocks to simultaneous reside in an SM. Since this is the same as the maximum allowed by the threading hardware, shared memory is not a limiting factor for this tile size. In this case, the real limitation is the threading hardware limitation that only 768 threads are allowed in each SM. This limits the number of blocks in each SM to three. As a result, only 3×2 KB=6 KB of the shared memory will be used. These limits do change from device generation to the next but are properties that can be determined at runtime, for example, the GT200 series of processors can support up to 1,024 threads in each SM.

请注意，每个 SM 中的共享内存大小也可能因设备而异。每一代或每一型号的设备在每个 SM 中可以具有不同数量的共享内存。通常希望内核能够根据硬件中的可用量使用不同数量的共享内存。也就是说，我们可能希望有一个内核来动态确定共享内存的大小并调整共享内存的使用量。这可以通过调用cudaGetDeviceProperties()函数来完成，其一般用法已在第 4.6 节中讨论。假设变量&dev_prop被传递给函数，字段dev_prop.sharedMemPerBlock给出了每个 SM 中可用的寄存器数量。然后，程序员可以确定每个块应使用的共享内存量。

Note that the size of shared memory in each SM can also vary from device to device. Each generation or model of device can have a different amount of shared memory in each SM. It is often desirable for a kernel to be able to use a different amount of shared memory according to the amount available in the hardware. That is, we may want to have a kernel to dynamically determine the size of the shared memory and adjust the amount of shared memory used. This can be done by calling the cudaGetDeviceProperties() function, the general use of which was discussed in Section 4.6. Assume that variable &dev_prop is passed to the function, the field dev_prop.sharedMemPerBlock gives the number of registers available in each SM. The programmer can then determine the number of amount of shared memory that should be used by each block.

不幸的是，图 5.12中的内核不支持这一点。图 5.12中使用的声明将其共享内存使用量的大小硬连接到编译时常量：

Unfortunately, the kernel in Figure 5.12 does not support this. The declarations used in Figure 5.12 hardwire the size of its shared memory usage to a compile-time constant:

 __shared__ float Mds[TILE_WIDTH][TILE_WIDTH];

 __shared__ float Mds[TILE_WIDTH][TILE_WIDTH];

 __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];

 __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];

也就是说，无论编译时TILE_WIDTH的值设置为多少，Mds和Nds的大小都设置为TILE_WIDTH ^{2 个}元素。例如，假设该文件包含

That is, the size of Mds and Nds is set to be TILE_WIDTH² elements, whatever the value of TILE_WIDTH is set to be at compile time. For example, assume that the file contains

 #定义TILE_WIDTH 16

 #define TILE_WIDTH 16

Mds和Nds都有 256 个元素。如果我们想改变Mds和Nds的大小，我们需要改变TILE_WIDTH的值并重新编译。如果不重新编译，内核无法在运行时轻松调整其共享内存使用情况。我们可以在 CUDA 中使用不同风格的声明来启用此类调整。我们可以在共享内存声明前面添加一个C extern关键字，并在声明中省略数组的大小。基于这种风格，Mds和Nds的声明变为：

Both Mds and Nds will have 256 elements. If we want to change the size of Mds and Nds, we need to change the value of TILE_WIDTH and recompile. The kernel cannot easily adjust its shared memory usage at runtime without recompilation. We can enable such adjustment with a different style of declaration in CUDA. We can add a C extern keyword in front of the shared memory declaration and omit the size of the array in the declaration. Based on this style, the declaration for Mds and Nds become:

 外部 __shared__ Mds[];

 extern __shared__ Mds[];

 外部 __shared__ Nds[];

 extern __shared__ Nds[];

请注意，数组现在是一维的。我们需要使用基于垂直和水平索引的线性化索引。

Note that the arrays are now one dimensional. We will need to use a linearized index based on the vertical and horizontal indices.

在运行时，当我们启动内核时，我们可以根据设备查询结果动态确定要使用的共享内存量，并将其作为第三个配置参数提供给内核发射。例如，图 4.18 中的内核启动语句可以替换为以下语句：

At runtime, when we launch the kernel, we can dynamically determine the amount of shared memory to be used according to the device query result and supply that as a third configuration parameter to the kernel launch. For example, the kernel launch statement in Figure 4.18 could be replaced with the following statements:

 size_t 尺寸 =

 size_t size =

  calculate_property_SM_usage(dev_prop.sharedMemPerBlock,…);

  calculate_appropriate_SM_usage(dev_prop.sharedMemPerBlock,…);

 矩阵MulKernel<<<dimGrid，dimBlock，尺寸>>>（Md，Nd，Pd，宽度）;

 matrixMulKernel<<<dimGrid, dimBlock, size>>>(Md, Nd, Pd, Width);

其中size_t是一个内置类型，用于声明一个变量来保存动态分配的数据结构的大小信息。我们省略了运行时设置 size 值的计算细节。

where size_t is a built-in type for declaring a variable to hold the size information for dynamically allocated data structures. We have omitted the details of the calculation for setting the value of size at runtime.

5.6 总结

5.6 Summary

总之，CUDA 定义了寄存器、共享内存和常量内存，它们可以比全局内存以更高的速度和更并行的方式进行访问。有效地使用这些存储器可能需要重新设计算法。我们以矩阵乘法为例来说明平铺算法，这是一种增强数据访问局部性并有效利用共享内存的流行策略。我们证明，通过 16×16 平铺，全局内存访问不再是矩阵乘法性能的主要限制因素。

In summary, CUDA defines registers, shared memory, and constant memory that can be accessed at a higher speed and in a more parallel manner than the global memory. Using these memories effectively will likely require redesign of the algorithm. We use matrix multiplication as an example to illustrate tiled algorithms, a popular strategy to enhance locality of data access and enable effective use of shared memory. We demonstrate that with 16×16 tiling, global memory accesses are no longer the major limiting factor for matrix multiplication performance.

然而，对于 CUDA 程序员来说，了解这些类型内存的有限大小非常重要。他们的能力取决于实施。一旦超出它们的容量，它们就会成为每个 SM 中可以同时执行的线程数量的限制因素。开发应用程序时推理硬件限制的能力是计算思维的一个关键方面。读者还可以参考附录 B，了解几种不同设备的资源限制摘要。

It is, however, important for CUDA programmers to be aware of the limited sizes of these types of memory. Their capacities are implementation dependent. Once their capacities are exceeded, they become limiting factors for the number of threads that can be simultaneously executing in each SM. The ability to reason about hardware limitations when developing an application is a key aspect of computational thinking. Readers are also referred to Appendix B for a summary of resource limitations of several different devices.

尽管我们在 CUDA 编程背景下引入了平铺算法，但它是在几乎所有类型的并行计算系统中实现高性能的有效策略。原因是应用程序必须在数据访问中表现出局部性，才能有效利用这些系统中的高速存储器。例如，在多核CPU系统中，数据局部性允许应用程序有效地使用片上数据缓存来减少内存访问延迟并实现高性能。因此，当读者使用其他编程模型为其他类型的并行计算系统开发并行应用程序时，会发现平铺算法很有用。

Although we introduced tiled algorithms in the context of CUDA programming, it is an effective strategy for achieving high performance in virtually all types of parallel computing systems. The reason is that an application must exhibit locality in data access to make effective use of high-speed memories in these systems. For example, in a multicore CPU system, data locality allows an application to effectively use on-chip data caches to reduce memory access latency and achieve high performance. Therefore, readers will find the tiled algorithm useful when they develop a parallel application for other types of parallel computing systems using other programming models.

本章的目标是介绍不同类型的 CUDA 内存。我们引入平铺算法作为使用共享内存的有效策略。我们还没有讨论常量内存的使用，这将在第 8 章中解释。

Our goal for this chapter is to introduce the different types of CUDA memory. We introduced tiled algorithm as an effective strategy for using shared memory. We have not discussed the use of constant memory, which will be explained in Chapter 8.

5.7 练习

5.7 Exercises

6.1 Warp 和线程执行

6.1 Warps and Thread Execution

我们首先讨论线程执行中可能限制性能的一些方面。回想一下，启动 CUDA 内核会生成一个组织为两级层次结构的线程网格。在顶层，网格由 1D、2D 或 3D 块阵列组成。在底层，每个块又由 1D、2D 或 3D 线程数组组成。在第 4 章中，我们看到块可以按彼此之间的任何顺序执行，这允许 CUDA 内核并行执行中的透明可扩展性。然而，我们并没有过多谈论每个块内线程的执行时序。

Let’s first discuss some aspects of thread execution that can limit performance. Recall that launching a CUDA kernel generates a grid of threads that are organized as a two-level hierarchy. At the top level, a grid consists of a 1D, 2D, or 3D array of blocks. At the bottom level, each block, in turn, consists of a 1D, 2D, or 3D array of threads. In Chapter 4, we saw that blocks can execute in any order relative to each other, which allows for transparent scalability in parallel execution of CUDA kernels. However, we did not say much about the execution timing of threads within each block.

从概念上讲，应该假设块中的线程可以按彼此之间的任何顺序执行。每当我们想要确保所有线程在任何线程开始下一阶段之前都完成了执行的公共阶段时，就应该使用屏障同步。执行内核的正确性不应依赖于某些线程彼此同步执行的事实。说到这里，我们还想指出，出于各种硬件成本的考虑，当前的CUDA设备实际上是捆绑了多个线程来执行的。这种实现策略导致某些类型的内核函数代码构造的性能限制。对于应用程序开发人员来说，将这些类型的构造更改为性能更好的其他等效形式是有利的。

Conceptually, one should assume that threads in a block can execute in any order with respect to each other. Barrier synchronizations should be used whenever we want to ensure all threads have completed a common phase of their execution before any of them start the next phase. The correctness of executing a kernel should not depend on the fact that certain threads will execute in synchrony with each other. Having said this, we also want to point out that due to various hardware cost considerations, current CUDA devices actually bundle multiple threads for execution. Such an implementation strategy leads to performance limitations for certain types of kernel function code constructs. It is advantageous for application developers to change these types of constructs to other equivalent forms that perform better.

正如我们在第 4 章中讨论的，当前的 CUDA 设备捆绑了多个执行线程。每个线程块都被划分为warp。 Warps 的执行由 SIMD 硬件实现（请参阅“Warps 和 SIMD 硬件”边栏）。这种实现技术有助于降低硬件制造成本、降低运行时操作电力成本，并在服务内存访问方面实现一些优化。在可预见的未来，我们预计扭曲分区仍将是一种流行的实现技术。然而，扭曲的大小很容易因实现的不同而不同。到目前为止，所有 CUDA 设备都使用类似的 warp 配置，其中每个 warp 由 32 个线程组成。

As we discussed in Chapter 4, current CUDA devices bundle several threads for execution. Each thread block is partitioned into warps. The execution of warps are implemented by an SIMD hardware (see “Warps and SIMD Hardware” sidebar). This implementation technique helps to reduce hardware manufacturing cost, lower runtime operation electricity cost, and enable some optimizations in servicing memory accesses. In the foreseeable future, we expect that warp partitioning will remain as a popular implementation technique. However, the size of a warp can easily vary from implementation to implementation. Up to this point in time, all CUDA devices have used similar warp configurations where each warp consists of 32 threads.

线程块根据线程索引划分为线程束。如果将线程块组织成一维数组（即仅使用threadIdx.x ），则分区很简单； warp 内的threadIdx.x值是连续且递增的。对于扭曲大小为 32，扭曲 0 从线程 0 开始，以线程 31 结束，扭曲 1 从线程 32 开始，以线程 63 结束。一般来说，扭曲n从线程 32× n开始，以线程 32( n +结束) 1)−1。对于大小不是 32 倍数的块，最后一个 warp 将用额外的线程填充以填充 32 个线程。例如，如果一个块有 48 个线程，它将被划分为两个 warp，并且其 warp 1 将用 16 个额外线程填充。

Thread blocks are partitioned into warps based on thread indices. If a thread block is organized into a 1D array (i.e., only threadIdx.x is used), the partition is straightforward; threadIdx.x values within a warp are consecutive and increasing. For a warp size of 32, warp 0 starts with thread 0 and ends with thread 31, warp 1 starts with thread 32 and ends with thread 63. In general, warp n starts with thread 32×n and ends with thread 32(n+1)−1. For a block of which the size is not a multiple of 32, the last warp will be padded with extra threads to fill up the 32 threads. For example, if a block has 48 threads, it will be partitioned into two warps, and its warp 1 will be padded with 16 extra threads.

对于由多个维度的线程组成的块，在划分为扭曲之前，维度将被投影为线性顺序。线性顺序是通过将y和z坐标较大的行放置在较小的行之后来确定的。也就是说，如果一个块由二维线程组成，则将所有threadIdx.y为1的线程放在threadIdx.y为0的线程之后，将threadIdx.y为2的线程放在那些线程之后，形成线性顺序。其中threadIdx.y为1，以此类推。

For blocks that consist of multiple dimensions of threads, the dimensions will be projected into a linear order before partitioning into warps. The linear order is determined by placing the rows with larger y and z coordinates after those with lower ones. That is, if a block consists of two dimensions of threads, one would form the linear order by placing all threads of which threadIdx.y is 1 after those of which threadIdx.y is 0, threads of which threadIdx.y is 2 after those of which threadIdx.y is 1, and so on.

图 6.1显示了将 2D 块的线程按线性顺序排列的示例。上半部分显示了块的 2D 视图。读者应该认识到与 C 语言中二维数组的行优先布局的相似性，如图4.3所示。每个线程显示为 T _y,x，x为threadIdx.x，y为threadIdx.y。图 6.1的下半部分显示了块的线性视图。前4个线程是threadIdx.y值为0的线程；它们按照递增的threadIdx.x值排序。接下来的四个线程是threadIdx.y值为1的线程；它们还被放置为递增的threadIdx.x值。对于此示例，所有 16 根纱线形成半个经纱。经纱将用另外 16 个线程填充，以完成 32 线程经纱。想象一个具有 8×8 线程的 2D 块。 64 根线程将形成两个经纱。第一个扭曲从 T _{0,0开始，以 T 3,7}结束。第二个扭曲从 T _4,0开始，到 T _7,7结束。画出图作为练习将是一个有用的练习。

Figure 6.1 shows an example of placing threads of a 2D block into linear order. The upper part shows the 2D view of the block. Readers should recognize the similarity with the row-major layout of 2D arrays in C, as shown in Figure 4.3. Each thread is shown as T_y,x, x being threadIdx.x and y being threadIdx.y. The lower part of Figure 6.1 shows the linear view of the block. The first four threads are those threads of which the threadIdx.y value is 0; they are ordered with increasing threadIdx.x values. The next four threads are those threads of which the threadIdx.y value is 1; they are also placed with increasing threadIdx.x values. For this example, all 16 threads form half a warp. The warp will be padded with another 16 threads to complete a 32-thread warp. Imagine a 2D block with 8×8 threads. The 64 threads will form two warps. The first warp starts from T_0,0 and ends with T_3,7. The second warp starts with T_4,0 and ends with T_7,7. It would be a useful exercise to draw out the picture as an exercise.

对于3D块，我们首先将threadIdx.z值为0的所有线程放入线性顺序。在这些线程中，它们被视为一个 2D 块，如图6.1所示。所有threadIdx.z值为1的线程将被放入线性顺序，依此类推。对于 3D尺寸为 2×8×4 的线程块（x维度有 4 个，y维度有 8 个，z维度有 2 个），64 个线程将被划分为两个 warp，其中 T _0,0,0到 T _{0 ,7,3}在第一经线中，T _1,0,0到 T _1,7,3在第二经线中。

For a 3D block, we first place all threads of which the threadIdx.z value is 0 into the linear order. Among these threads, they are treated as a 2D block as shown in Figure 6.1. All threads of which the threadIdx.z value is 1 will then be placed into the linear order, and so on. For a 3D thread block of dimensions 2×8×4 (four in the x dimension, eight in the y dimension, and two in the z dimension), the 64 threads will be partitioned into two warps, with T_0,0,0 through T_0,7,3 in the first warp and T_1,0,0 through T_1,7,3 in the second warp.

SIMD 硬件将 warp 的所有线程作为一个包执行。一条指令针对同一经束中的所有线程运行。当扭曲中的所有线程在处理数据时遵循相同的执行路径（或更正式地称为控制流）时，它会很好地工作。例如，对于if-else构造，当所有线程都执行if部分或全部执行else部分时，执行效果良好。当 warp 内的线程采用不同的控制流路径时，SIMD 硬件将多次通过这些不同的路径。一次传递执行if部分后面的线程，另一遍执行else部分后面的线程。在每次传递期间，不允许遵循其他路径的线程生效。这些遍是彼此连续的，因此它们会增加执行时间。

The SIMD hardware executes all threads of a warp as a bundle. An instruction is run for all threads in the same warp. It works well when all threads within a warp follow the same execution path, or more formally referred to as control flow, when working their data. For example, for an if-else construct, the execution works well when either all threads execute the if part or all execute the else part. When threads within a warp take different control flow paths, the SIMD hardware will take multiple passes through these divergent paths. One pass executes those threads that follow the if part and another pass executes those that follow the else part. During each pass, the threads that follow the other path are not allowed to take effect. These passes are sequential to each other, thus they will add to the execution time.

发散扭曲执行的多通道方法扩展了 SIMD 硬件实现 CUDA 线程完整语义的能力。虽然硬件对 warp 中的所有线程执行相同的指令，但它有选择地让线程仅在每次传递中生效，从而允许每个线程采用自己的控制流路径。这保留了线程的独立性，同时利用了 SIMD 硬件成本降低的优势。

The multipass approach to divergent warp execution extends the SIMD hardware’s ability to implement the full semantics of CUDA threads. While the hardware executes the same instruction for all threads in a warp, it selectively lets the threads take effect in each pass only, allowing every thread to take its own control flow path. This preserves the independence of threads while taking advantage of the reduced cost of SIMD hardware.

当同一 warp 中的线程遵循不同的控制流路径时，我们称这些线程在执行中出现分歧。在if-else示例中，如果经束中的某些线程采用then路径而另一些线​​程采用else路径，则会出现分歧。发散的成本是硬件需要采取额外的传递以允许扭曲中的线程做出自己的决定。其他结构中也可能出现分歧；例如，如果 warp 中的线程执行for循环，则可以为不同的线程迭代六次、七次或八次。所有线程将一起完成前六次迭代。将使用两次迭代来执行第七次迭代，一次用于进行迭代的迭代，一次用于不进行迭代的迭代。将使用两次迭代来执行第八次迭代，一次用于进行迭代的迭代，一次用于不进行迭代的迭代。

When threads in the same warp follow different paths of control flow, we say that these threads diverge in their execution. In the if-else example, divergence arises if some threads in a warp take the then path and some the else path. The cost of divergence is the extra pass the hardware needs to take to allow the threads in a warp to make their own decisions. Divergence also can arise in other constructs; for example, if threads in a warp execute a for loop that can iterate six, seven, or eight times for different threads. All threads will finish the first six iterations together. Two passes will be used to execute the seventh iteration, one for those that take the iteration and one for those that do not. Two passes will be used to execute the eighth iteration, one for those that take the iteration and one for those that do not.

就源语句而言，当其决策条件基于threadIdx值时，控制构造可能会导致线程发散。例如，语句if (threadIdx.x > 2) {}导致线程遵循两个不同的控制流路径。线程 0、1 和 2 遵循与线程 3、4、5 等不同的路径。类似地，循环可能会导致线程如果其循环条件基于线程索引值，则发散。这种用法自然出现在一些重要的并行算法中。我们将使用归约算法来说明这一点。

In terms of source statements, a control construct can result in thread divergence when its decision condition is based on threadIdx values. For example, the statement if (threadIdx.x > 2) {} causes the threads to follow two divergent control flow paths. Threads 0, 1, and 2 follow a different path than threads 3, 4, 5, etc. Similarly, a loop can cause thread divergence if its loop condition is based on thread index values. Such usages arise naturally in some important parallel algorithms. We will use a reduction algorithm to illustrate this point.

归约算法从值数组中导出单个值。单个值可以是所有元素之间的总和、最大值、最小值等。所有这些类型的归约共享相同的计算结构。通过按顺序遍历数组的每个元素可以轻松完成缩减。当访问一个元素时，要采取的操作取决于所执行的归约类型。对于总和减少，当前步骤中访问的元素的值或当前值被添加到运行总和中。对于最大减少，将当前值与迄今为止访问的所有元素的运行最大值进行比较。如果当前值大于运行最大值，则当前元素值将变为运行最大值。对于最小减少，当前正在访问的元素的值与运行最小值进行比较。如果当前值小于运行最小值，则当前元素值变为运行最小值。当所有元素都被访问完时，顺序算法结束。顺序归约算法的工作效率很高，因为每个元素仅被访问一次，并且在访问每个元素时仅执行最少量的工作。其执行时间与涉及的元素数量成正比。也就是说，该算法的计算复杂度为O ( N )，其中N是参与归约的元素数量。

A reduction algorithm derives a single value from an array of values. The single value could be the sum, the maximal value, the minimal value, etc. among all elements. All these types of reductions share the same computation structure. A reduction can be easily done by sequentially going through every element of the array. When an element is visited, the action to take depends on the type of reduction being performed. For a sum reduction, the value of the element being visited at the current step, or the current value, is added to a running sum. For a maximal reduction, the current value is compared to a running maximal value of all the elements visited so far. If the current value is larger than the running maximal, the current element value becomes the running maximal value. For a minimal reduction, the value of the element currently being visited is compared to a running minimal. If the current value is smaller than the running minimal, the current element value becomes the running minimal. The sequential algorithm ends when all the elements are visited. The sequential reduction algorithm is work-efficient in that every element is only visited once and only a minimal amount of work is performed when each element is visited. Its execution time is proportional to the number of elements involved. That is, the computational complexity of the algorithm is O(N), where N is the number of elements involved in the reduction.

访问大型数组的所有元素所需的时间激发了并行执行。并行缩减算法通常类似于足球锦标赛的结构。事实上，世界杯的淘汰过程就是“最大”的缩减，最大的定义是“击败”所有其他球队的球队。比赛“减少”是通过多轮进行的。各队被分成两人一组。在第一轮比赛中，所有双人组并行比赛。第一轮获胜者晋级第二轮，第二轮获胜者晋级第三轮，以此类推。 16支球队参赛，第一轮获胜者为8名，第二轮获胜者为4名，第三轮获胜者为2名，以及第四轮的 1 名最终获胜者。应该很容易看出，即使有1024支球队，也只需要10轮就可以确定最终的获胜者。诀窍是要有足够的足球场来同时举办第一轮的512场比赛、第二轮的256场比赛、第三轮的128场比赛，以此类推。在场地足够的情况下，即使有6万支球队，我们也只需16轮就可以决出最后的胜者​​。当然，前提是要有足够的足球场和足够的官员来容纳第一轮的 30,000 场比赛，等等。

The time needed to visit all elements of a large array motivates parallel execution. A parallel reduction algorithm typically resembles the structure of a soccer tournament. In fact, the elimination process of the World Cup is a reduction of “maximal” where the maximal is defined as the team that “beats” all other teams. The tournament “reduction” is done by multiple rounds. The teams are divided into pairs. During the first round, all pairs play in parallel. Winners of the first round advance to the second round, the winners of which advance to the third round, etc. With 16 teams entering a tournament, 8 winners will emerge from the first round, 4 from the second round, 2 from the third round, and 1 final winner from the fourth round. It should be easy to see that even with 1,024 teams, it takes only 10 rounds to determine the final winner. The trick is to have enough soccer fields to hold the 512 games in parallel during the first round, 256 games in the second round, 128 games in the third round, and so on. With enough fields, even with 60,000 teams, we can determine the final winner in just 16 rounds. Of course, one would need to have enough soccer fields and enough officials to accommodate the 30,000 games in the first round, etc.

图 6.2显示了执行并行求和缩减的核函数。原始数组位于全局内存中。每个线程块通过将部分元素加载到共享内存中并执行并行缩减来缩减数组的一部分。为了简洁起见，图 6.2中省略了将元素从全局内存加载到共享内存的代码。减少是就地完成的，这意味着共享内存中的元素将被部分和替换。核函数中while循环的每次迭代都会实现一轮归约。while循环中的__syncthreads()语句（第 5 行）确保已生成前一迭代的所有部分和，因此所有线程都已准备好进入当前迭代，然后才允许其中任何一个线程这样做。这样，进入第二次迭代的所有线程都将使用第一次迭代中生成的值。第一轮结束后，偶数元素将被第一轮生成的部分和替换。第二轮结束后，索引为四的倍数的元素将被部分和替换。最后一轮结束后，整个部分的总和将位于元素 0 中。

Figure 6.2 shows a kernel function that performs parallel sum reduction. The original array is in the global memory. Each thread block reduces a section of the array by loading the elements of the section into the shared memory and performing parallel reduction. The code that loads the elements from global memory into the shared memory is omitted from Figure 6.2 for brevity. The reduction is done in place, which means the elements in the shared memory will be replaced by partial sums. Each iteration of the while loop in the kernel function implements a round of reduction. The __syncthreads() statement (line 5) in the while loop ensures that all partial sums for the previous iteration have been generated and thus all threads are ready to enter the current iteration before any one of them is allowed to do so. This way, all threads that enter the second iteration will be using the values produced in the first iteration. After the first round, the even elements will be replaced by the partial sums generated in the first round. After the second round, the elements of which the indices are multiples of four will be replaced with the partial sums. After the final round, the total sum of the entire section will be in element 0.

在图 6.2中，第 3 行将 stride 变量初始化为 1。在第一次迭代期间，第 6 行中的if语句用于仅选择偶数线程来执行两个相邻元素之间的加法。内核的执行如图6.3所示。线程和数组元素值以水平方向显示。线程所进行的迭代以垂直方向显示，时间从上到下进行。图 6.3的每一行显示了for循环迭代后数组元素的内容。

In Figure 6.2, line 3 initializes the stride variable to 1. During the first iteration, the if statement in line 6 is used to select only the even threads to perform addition between two neighboring elements. The execution of the kernel is illustrated in Figure 6.3. The threads and the array element values are shown in the horizontal direction. The iterations taken by the threads are shown in the vertical direction with time progressing from top to bottom. Each row of Figure 6.3 shows the contents of the array elements after an iteration of the for loop.

如图6.3所示，数组的偶数元素保存第 1 次迭代后的成对部分和。在第二次迭代之前，stride变量的值加倍为 2。在第二次迭代期间，仅那些索引为四的倍数将执行第 8 行中的add语句。每个线程生成包含四个元素的部分和，如第 2 行所示。每个部分有 512 个元素，内核函数将在九次迭代后生成整个部分的和。通过使用blockDim.x作为第 4 行中的循环绑定，内核假定它是使用与该部分中的元素数量相同数量的线程启动的。也就是说，对于 512 的部分大小，需要使用 512 个线程启动内核。¹

As shown in Figure 6.3, the even elements of the array hold the pairwise partial sums after iteration 1. Before the second iteration, the value of the stride variable is doubled to 2. During the second iteration, only those threads of which the indices are multiples of four will execute the add statement in line 8. Each thread generates a partial sum that includes four elements, as shown in row 2. With 512 elements in each section, the kernel function will generate the sum of the entire section after nine iterations. By using blockDim.x as the loop bound in line 4, the kernel assumes that it is launched with the same number of threads as the number of elements in the section. That is, for a section size of 512, the kernel needs to be launched with 512 threads.¹

我们来分析一下内核完成的总工作量。假设要减少的元素总数为N。第一轮需要N /2次加法。第二轮需要N /4个加法。最后一轮只有一个补充。有 log ₂ ( N ) 轮。内核执行的加法总数为N /2+ N /4+ N /8+…+1= N −1。因此，约简算法的计算复杂度为O ( N )。该算法工作效率高。然而，我们还需要确保在执行内核时有效地利用硬件。

Let’s analyze the total amount of work done by the kernel. Assume that the total number of elements to be reduced is N. The first round requires N/2 additions. The second round requires N/4 additions. The final round has only one addition. There are log₂(N) rounds. The total number of additions performed by the kernel is N/2+N/4+N/8+…+1=N−1. Therefore, the computational complexity of the reduction algorithm is O(N). The algorithm is work-efficient. However, we also need to make sure that the hardware is efficiently utilized while executing the kernel.

图 6.2中的内核明显存在线程分歧。在循环的第一次迭代期间，只有那些threadIdx.x为偶数的线程才会执行add语句。需要一次传递来执行这些线程，并且需要额外一次传递来执行那些不执行第 8 行的线程。在每次连续迭代中，执行第 8 行的线程会减少，但仍需要两次传递来执行期间的所有线程。每次迭代。通过稍微改变算法就可以减少这种差异。

The kernel in Figure 6.2 clearly has thread divergence. During the first iteration of the loop, only those threads of which the threadIdx.x are even will execute the add statement. One pass will be needed to execute these threads and one additional pass will be needed to execute those that do not execute line 8. In each successive iteration, fewer threads will execute line 8 but two passes will be still needed to execute all the threads during each iteration. This divergence can be reduced with a slight change to the algorithm.

图 6.4显示了修改后的内核，其求和缩减算法略有不同。它不是在第一轮中添加相邻元素，而是添加彼此相距半个截面的元素。它通过将步幅初始化为该部分大小的一半来实现这一点。在第一轮中添加的所有对彼此之间的距离是部分大小的一半。第一次迭代后，所有成对和都存储在数组的前半部分。该循环在进入下一次迭代之前将步幅除以 2。因此，对于第二次迭代，步幅变量值是部分大小的四分之一，也就是说，线程在第二次迭代期间添加彼此相距四分之一部分的元素。

Figure 6.4 shows a modified kernel with a slightly different algorithm for sum reduction. Instead of adding neighbor elements in the first round, it adds elements that are half a section away from each other. It does so by initializing the stride to be half the size of the section. All pairs added during the first round are half the section size away from each other. After the first iteration, all the pairwise sums are stored in the first half of the array. The loop divides the stride by 2 before entering the next iteration. Thus, for the second iteration, the stride variable value is one-quarter of the section size—that is, the threads add elements that are one-quarter a section away from each other during the second iteration.

请注意，图 6.4中的内核在循环中仍然有一个if语句（第 6 行）。每次迭代中执行第 7 行的线程数量与图 6.2中的相同。那么，为什么两个内核之间会有性能差异呢？答案在于执行第 7 行的线程相对于不执行第 7 行的线程的位置。

Note that the kernel in Figure 6.4 still has an if statement (line 6) in the loop. The number of threads that execute line 7 in each iteration is the same as in Figure 6.2. So, why should there be a performance difference between the two kernels? The answer lies in the positions of threads that execute line 7 relative to those that do not.

图 6.5说明了修改后的内核的执行情况。在第一次迭代期间，threadIdx.x值小于该节大小一半的所有线程都执行第 7 行。对于 512 个元素的节，线程 0–255在第一次迭代期间执行add语句而线程 256-511 则不然。第一次迭代后，成对的和存储在元素 0-255 中。由于 warps 由 32 个具有连续threadIdx.x值的线程组成，warps 0-7 中的所有线程都执行 add 语句，而 warps 8-15 都跳过 add 语句。由于每个经纱中的所有线程都采用相同的路径，因此不存在线程分歧！

Figure 6.5 illustrates the execution of the revised kernel. During the first iteration, all threads of which the threadIdx.x values are less than half of the size of the section execute line 7. For a section of 512 elements, threads 0–255 execute the add statement during the first iteration while threads 256–511 do not. The pairwise sums are stored in elements 0–255 after the first iteration. Since the warps consist of 32 threads with consecutive threadIdx.x values, all threads in warps 0–7 execute the add statement, whereas warps 8–15 all skip the add statement. Since all threads in each warp take the same path, there is no thread divergence!

图 6.4中的内核并没有完全消除if语句导致的分歧。读者应该验证从第五次迭代开始，执行第 7 行的线程数量将降至 32 以下。也就是说，最后五次迭代将只有 16、8、4、2 和 1 个线程执行加法。这意味着内核执行在这些迭代中仍然会存在分歧。然而，有分歧的循环迭代次数从 10 次减少到 5 次。

The kernel in Figure 6.4 does not completely eliminate the divergence due to the if statement. Readers should verify that starting with the fifth iteration, the number of threads that execute line 7 will fall below 32. That is, the final five iterations will have only 16, 8, 4, 2, and 1 thread(s) performing the addition. This means that the kernel execution will still have divergence in these iterations. However, the number of iterations of the loop that has divergence is reduced from 10 to 5.

6.2 全局内存带宽

6.2 Global Memory Bandwidth

CUDA 内核性能最重要的因素之一是访问全局内存中的数据。 CUDA 应用程序利用海量数据并行性。当然，CUDA 应用程序倾向于在短时间内处理全局内存中的大量数据。在第 5 章中，我们讨论了利用共享内存来减少线程块中的线程集合必须访问的数据总量的平铺技术。在本章中，我们将进一步讨论内存合并技术可以更有效地将数据从全局内存移动到共享内存和寄存器中。内存合并技术通常与切片技术结合使用，以允许 CUDA 设备通过更有效地利用全局内存带宽来发挥其性能潜力。²

One of the most important factors of CUDA kernel performance is accessing data in the global memory. CUDA applications exploit massive data parallelism. Naturally, CUDA applications tend to process a massive amount of data from the global memory within a short period of time. In Chapter 5, we discussed tiling techniques that utilize shared memories to reduce the total amount of data that must be accessed by a collection of threads in the thread block. In this chapter, we will further discuss memory coalescing techniques that can more effectively move data from the global memory into shared memories and registers. Memory coalescing techniques are often used in conjunction with tiling techniques to allow CUDA devices to reach their performance potential by more efficiently utilizing the global memory bandwidth.²

CUDA 设备的全局内存是通过 DRAM 实现的。数据位存储在 DRAM 单元中，这些单元是小型电容器，其中存在或不存在微小电荷可区分 0 和 1。从 DRAM 单元读取数据需要小型电容器使用其微小电荷来驱动高电压。电容线通向传感器并启动其检测机制，确定电容器中是否存在足够的电荷量以符合“1”的条件（请参阅“为什么 DRAM 如此慢？”侧边栏）。在现代 DRAM 芯片中，这个过程需要数十纳秒。由于相对于所需的数据访问速度（每字节亚纳秒访问）而言，这是一个非常慢的过程，因此现代 DRAM 使用并行性来提高数据访问速率。

The global memory of a CUDA device is implemented with DRAMs. Data bits are stored in DRAM cells that are small capacitors, where the presence or absence of a tiny amount of electrical charge distinguishes between 0 and 1. Reading data from a DRAM cell requires the small capacitor to use its tiny electrical charge to drive a highly capacitive line leading to a sensor and set off its detection mechanism that determines whether a sufficient amount of charge is present in the capacitor to qualify as a “1” (see “Why Are DRAMs So Slow?” sidebar). This process takes tens of nanoseconds in modern DRAM chips. Because this is a very slow process relative to the desired data access speed (sub-nanosecond access per byte), modern DRAMs use parallelism to increase their rate of data access.

每次访问 DRAM 位置时，实际上都会访问包括所请求位置的许多连续位置。每个 DRAM 芯片中都设有许多传感器，它们并行工作。每个都感测这些连续位置内的一位的内容。一旦被传感器检测到，来自所有这些连续位置的数据就可以以非常高的速度传输到处理器。如果应用程序可以集中使用来自连续位置的数据，那么 DRAM 可以以比访问真正随机的位置序列高得多的速率提供数据。

Each time a DRAM location is accessed, many consecutive locations that include the requested location are actually accessed. Many sensors are provided in each DRAM chip and they work in parallel. Each senses the content of a bit within these consecutive locations. Once detected by the sensors, the data from all these consecutive locations can be transferred at very high speed to the processor. If an application can make focused use of data from consecutive locations, the DRAMs can supply the data at a much higher rate than if a truly random sequence of locations were accessed.

认识到现代 DRAM 的组织方式，当前的 CUDA 设备采用了一种技术，允许程序员通过将线程的内存访问组织成有利的模式来实现高全局内存访问效率。该技术利用了扭曲中的线程在任何给定时间点执行相同指令的事实。当 warp 中的所有线程执行加载指令时，硬件会检测它们是否访问连续的全局内存位置。也就是说，当线程束中的所有线程访问连续的全局内存位置时，实现最有利的访问模式。在这种情况下，硬件将所有这些访问组合或合并为对连续 DRAM 位置的合并访问。例如，对于给定的 warp 加载指令，如果线程 0 访问全局内存位置N、^{3 个}线程 1 位置N +1 、线程 2 位置N +2 等等，则所有这些访问将被合并或组合成访问 DRAM 时对连续位置的单个请求。这种联合访问允许 DRAM 以接近峰值全局内存带宽的速率传送数据。

Recognizing the organization of modern DRAMs, current CUDA devices employ a technique that allows the programmers to achieve high global memory access efficiency by organizing memory accesses of threads into favorable patterns. This technique takes advantage of the fact that threads in a warp execute the same instruction at any given point in time. When all threads in a warp execute a load instruction, the hardware detects whether they access consecutive global memory locations. That is, the most favorable access pattern is achieved when all threads in a warp access consecutive global memory locations. In this case, the hardware combines, or coalesces, all these accesses into a consolidated access to consecutive DRAM locations. For example, for a given load instruction of a warp, if thread 0 accesses global memory location N,³ thread 1 location N+1, thread 2 location N+2, and so on, all these accesses will be coalesced, or combined into a single request for consecutive locations when accessing the DRAMs. Such coalesced access allows the DRAMs to deliver data at a rate close to the peak global memory bandwidth.

为了理解如何有效地使用合并硬件，我们需要回顾一下在访问 C 时内存地址是如何形成的多维数组元素。正如我们在第 4 章中所示（图 4.3，为了方便起见，复制为图 6.6），C 和 CUDA 中的多维数组元素根据行优先约定放置到线性寻址内存空间中。也就是说，矩阵第 0 行的元素首先按顺序放置到连续的位置。它们后面是矩阵第 1 行的元素，依此类推。换句话说，一行中的所有元素都被放置到连续的位置，并且整行被一个接一个地放置。术语“行主”是指数据的放置保留了行的结构：行中的所有相邻元素都被放置到地址空间中的连续位置。图 6.6显示了一个小示例，其中 4×4 矩阵 M 的 16 个元素被放置到线性寻址位置。第 0 行的四个元素首先按照其在该行中出现的顺序放置。然后放置第 1 行中的元素，然后是第 2 行的元素，最后是第 3 行的元素。应该清楚的是，尽管 M _0,0和 M _1,0在 2D 矩阵中看起来是连续的，但它们是被放置的线性寻址存储器中的四个位置相距。

To understand how to effectively use coalescing hardware, we need to review how the memory addresses are formed in accessing C multidimensional array elements. As we showed in Chapter 4 (Figure 4.3, replicated as Figure 6.6 for convenience), multidimensional array elements in C and CUDA are placed into the linearly addressed memory space according to the row-major convention. That is, the elements of row 0 of a matrix are first placed in order into consecutive locations. They are followed by the elements of row 1 of the matrix, and so on. In other words, all elements in a row are placed into consecutive locations and entire rows are placed one after another. The term row major refers to the fact that the placement of data preserves the structure of rows: all adjacent elements in a row are placed into consecutive locations in the address space. Figure 6.6 shows a small example where the 16 elements of a 4×4 matrix M are placed into linearly addressed locations. The four elements of row 0 are first placed in their order of appearance in the row. Elements in row 1 are then placed, followed by elements of row 2, followed by elements of row 3. It should be clear that M_0,0 and M_1,0, though they appear to be consecutive in the 2D matrix, are placed four locations away in the linearly addressed memory.

图 6.7说明了内存合并的有利与不利的 CUDA 内核 2D 行主数组数据访问模式。回想一下图 4.7 ，在我们的简单矩阵-矩阵乘法内核中，每个线程访问d_M数组的一行和d_N数组的一列。读者在继续之前应先阅读第 4.3 节。图 6.7(a)说明了d_M数组的数据访问模式，其中 warp 中的线程读取相邻行。也就是说，在迭代 0 期间，warp 中的线程读取第 0-31 行的元素 0。在迭代 1 期间，这些相同的线程读取第 0-31 行的元素 1。所有访问都不会被合并。更有利的访问模式如图 6.7(b)所示，其中每个线程读取一列d_N 。​在迭代 0 期间，warp 0 中的线程读取列 0–31 的元素 1。所有这些访问都将被合并。

Figure 6.7 illustrates the favorable versus unfavorable CUDA kernel 2D row-major array data access patterns for memory coalescing. Recall from Figure 4.7 that in our simple matrix–matrix multiplication kernel, each thread accesses a row of the d_M array and a column of the d_N array. Readers should review Section 4.3 before continuing. Figure 6.7(a) illustrates the data access pattern of the d_M array, where threads in a warp read adjacent rows. That is, during iteration 0, threads in a warp read element 0 of rows 0–31. During iteration 1, these same threads read element 1 of rows 0–31. None of the accesses will be coalesced. A more favorable access pattern is shown in Figure 6.7(b), where each thread reads a column of d_N. During iteration 0, threads in warp 0 read element 1 of columns 0–31. All these accesses will be coalesced.

为了理解为什么图 6.7(b)中的模式比图 6.7(a)中的模式更有利，我们需要更详细地回顾一下如何访问这些矩阵元素。图 6.8显示了访问 4×4 矩阵时有利的访问模式的一个小示例。图 6.8顶部的箭头显示了一个线程的内核代码的访问模式。该访问模式是通过访问图 4.7中的d_N生成的：

To understand why the pattern in Figure 6.7(b) is more favorable than that in Figure 6.7(a), we need to review how these matrix elements are accessed in more detail. Figure 6.8 shows a small example of the favorable access pattern in accessing a 4×4 matrix. The arrow in the top portion of Figure 6.8 shows the access pattern of the kernel code for one thread. This access pattern is generated by the access to d_N in Figure 4.7:

d_N[k*宽度 + 列]

d_N[k∗Width + Col]

在k循环的给定迭代内，所有线程的k*Width值都相同。回想一下Col = blockIdx.x*blockDim.x + threadIdx.x。由于blockIndx.x和blockDim.x的值对于同一块中的所有线程具有相同的值，因此k*Width+Col在线程块中唯一变化的部分是threadIdx.x。例如，在图 6.8中，假设我们使用 4×4 块并且扭曲大小为 4。也就是说，对于这个玩具示例，我们仅使用一个块来计算整个 P 矩阵。对于块中的所有线程， Width、blockDim.x和blockIdx.x的值分别为 4、4 和 0。在迭代0中，k值为0。每个线程访问d_N使用的索引为

Within a given iteration of the k loop, the k∗Width value is the same across all threads. Recall that Col = blockIdx.x∗blockDim.x + threadIdx.x. Since the value of blockIndx.x and blockDim.x are of the same value for all threads in the same block, the only part of k∗Width+Col that varies across a thread block is threadIdx.x. For example, in Figure 6.8, assume that we are using 4×4 blocks and that the warp size is 4. That is, for this toy example, we are using only one block to calculate the entire P matrix. The values of Width, blockDim.x, and blockIdx.x are 4, 4, and 0, respectively, for all threads in the block. In iteration 0, the k value is 0. The index used by each thread for accessing d_N is

 d_N[k*Width+Col]=d_N[k*Width+blockIdx.x*blockDim.x+threadIdx.x]

 d_N[k∗Width+Col]=d_N[k∗Width+blockIdx.x∗blockDim.x+threadIdx.x]

 = d_N[0*4 + 0*4 + threadidx.x]

 = d_N[0∗4 + 0∗4 + threadidx.x]

 = d_N[线程Idx.x]

 = d_N[threadIdx.x]

也就是说，访问d_N 的索引就是threadIdx.x的值。 T ₀、T ₁、T ₂和T ₃访问的d_N元素分别是d_N[0]、d_N[1]、d_N[2]和d_N[3]。图 6.8的“加载迭代 0”框对此进行了说明。这些元素位于全局内存中的连续位置。硬件检测到这些访问是由线程束中的线程对全局内存中的连续位置进行的。它将这些访问合并为一个合并的访问。这使得 DRAM 能够以高速率提供数据。

That is, the index for accessing d_N is simply the value of threadIdx.x. The d_N elements accessed by T₀, T₁, T₂, and T₃ are d_N[0], d_N[1], d_N[2], and d_N[3], respectively. This is illustrated with the “Load iteration 0” box of Figure 6.8. These elements are in consecutive locations in the global memory. The hardware detects that these accesses are made by threads in a warp and to consecutive locations in the global memory. It coalesces these accesses into a consolidated access. This allows the DRAMs to supply data at a high rate.

在下一次迭代中，k值为1。每个线程访问d_N所使用的索引变为

During the next iteration, the k value is 1. The index used by each thread for accessing d_N becomes

 d_N[k*Width+Col]=d_N[k*Width+blockIdx.x*blockDim.x+threadIdx.x]

 d_N[k∗Width+Col]=d_N[k∗Width+blockIdx.x∗blockDim.x+threadIdx.x]

 = d_N[1*4 + 0*4 + threadidx.x]

 = d_N[1∗4 + 0∗4 + threadidx.x]

 = d_N[4+threadIdx.x]

 = d_N[4+threadIdx.x]

_{T 0}、 T ₁、 T ₂和 T ₃访问的d_N元素分别是d_N[5]、d_N[6]、d_N[7]和d_N[8]，如“加载迭代 1”所示图 6.8中的方框。所有这些访问再次合并为一个综合访问，以提高 DRAM 带宽利用率。

The d_N elements accessed by T₀, T₁, T₂, and T₃ are d_N[5], d_N[6], d_N[7], and d_N[8], respectively, as shown with the “Load iteration 1” box in Figure 6.8. All these accesses are again coalesced into a consolidated access for improved DRAM bandwidth utilization.

图 6.9显示了未合并的矩阵数据访问模式的示例。该图顶部的箭头显示每个线程的内核代码按顺序访问一行的元素。图 6.9顶部的箭头显示了一个线程的内核代码的访问模式。该访问模式是通过访问图 4.7中的d_M生成的：

Figure 6.9 shows an example of a matrix data access pattern that is not coalesced. The arrow in the top portion of the figure shows that the kernel code for each thread accesses elements of a row in sequence. The arrow in the top portion of Figure 6.9 shows the access pattern of the kernel code for one thread. This access pattern is generated by the access to d_M in Figure 4.7:

 d_M[行*宽度 + k]

 d_M[Row∗Width + k]

在k循环的给定迭代内，所有线程的k*Width值都相同。回想一下Row = blockIdx.y*blockDim.y + 线程Idx.y。由于blockIndx.y和blockDim.y的值对于同一块中的所有线程具有相同的值，因此Row*Width+k唯一可以在线程块中变化的部分是threadIdx.y。在图 6.9中，再次假设我们使用 4×4 块，并且扭曲大小为 4。对于该块中的所有线程， Width、blockDim.y和blockIdx.y的值分别为 4、4 和 0。堵塞。在迭代0中，k值为0。每个线程访问d_N使用的索引为

Within a given iteration of the k loop, the k∗Width value is the same across all threads. Recall that Row = blockIdx.y∗blockDim.y + threadIdx.y. Since the value of blockIndx.y and blockDim.y are of the same value for all threads in the same block, the only part of Row∗Width+k that can vary across a thread block is threadIdx.y. In Figure 6.9, assume again that we are using 4×4 blocks and that the warp size is 4. The values of Width, blockDim.y, and blockIdx.y are 4, 4, and 0, respectively, for all threads in the block. In iteration 0, the k value is 0. The index used by each thread for accessing d_N is

 d_M[行*宽度+k] = d_M[(blockIdx.y*blockDim.y+threadIdx.y)*宽度+k]

 d_M[Row∗Width+k] = d_M[(blockIdx.y∗blockDim.y+threadIdx.y)∗Width+k]

 = d_M[((0*4+threadIdx.y)*4 + 0]

 = d_M[((0∗4+threadIdx.y)∗4 + 0]

 = d_M[threadIdx.x*4]

 = d_M[threadIdx.x∗4]

也就是说，访问d_M的索引就是threadIdx.x*4的值。 T ₀、T ₁、T ₂和T ₃访问的d_M元素是d_M[0]、d_M[4]、d_M[8]和d_M[12]。图 6.9的“加载迭代 0”框对此进行了说明。这些元素不在全局内存中连续的位置。硬件无法将这些访问合并为合并的访问。

That is, the index for accessing d_M is simply the value of threadIdx.x∗4. The d_M elements accessed by T₀, T₁, T₂, and T₃ are d_M[0], d_M[4], d_M[8], and d_M[12]. This is illustrated with the “Load iteration 0” box of Figure 6.9. These elements are not in consecutive locations in the global memory. The hardware cannot coalesce these accesses into a consolidated access.

在下一次迭代中，k值为1。每个线程访问d_M所使用的索引变为

During the next iteration, the k value is 1. The index used by each thread for accessing d_M becomes

 d_M[行*宽度+k] = d_M[(blockIdx.y*blockDim.y+threadIdx.y)*宽度+k]

 d_M[Row∗Width+k] = d_M[(blockIdx.y∗blockDim.y+threadIdx.y)∗Width+k]

 = d_M[(0*4+threadidx.x)*4+1]

 = d_M[(0∗4+threadidx.x)∗4+1]

 = d_M[threadIdx.x*4+1]

 = d_M[threadIdx.x∗4+1]

_{T 0}、 T ₁、 T ₂、 T ₃访问的d_M元素分别为d_M[1]、d_M[5]、d_M[9]和d_M[13]，如“加载迭代 1”框所示如图 6.9所示。所有这些访问再次无法合并为合并访问。

The d_M elements accessed by T₀, T₁, T₂, T₃ are d_M[1], d_M[5], d_M[9], and d_M[13], respectively, as shown with the “Load iteration 1” box in Figure 6.9. All these accesses again cannot be coalesced into a consolidated access.

对于实际矩阵，每个维度通常有数百甚至数千个元素。相邻线程在每次迭代中访问的元素可能相距数百甚至数千个元素。底部的“加载迭代 0”框显示了线程如何在 0 迭代中访问这些不连续的位置。硬件将确定对这些元素的访问彼此相距较远并且无法合并。因此，当内核循环迭代行时，对全局内存的访问效率比内核迭代列的情况低得多。

For a realistic matrix, there are typically hundreds or even thousands of elements in each dimension. The elements accessed in each iteration by neighboring threads can be hundreds or even thousands of elements apart. The “Load iteration 0” box in the bottom portion shows how the threads access these nonconsecutive locations in the 0 iteration. The hardware will determine that accesses to these elements are far away from each other and cannot be coalesced. As a result, when a kernel loop iterates through a row, the accesses to global memory are much less efficient than the case where a kernel iterates through a column.

如果算法本质上需要内核代码沿行方向迭代数据，则可以使用共享内存来启用内存合并。图 6.10说明了矩阵乘法的技术。每个线程从d_M读取一行，这是一种不能被读取的模式合并。幸运的是，可以使用平铺算法来启用合并。正如我们在第 5 章中讨论的，块的线程可以首先协作地将图块加载到共享内存中。必须小心确保这些瓷砖以合并的模式加载。一旦数据位于共享存储器中，就可以按行或列进行访问，性能变化要小得多，因为共享存储器被实现为本质上高速的片上存储器，不需要合并来实现高数据访问率。

If an algorithm intrinsically requires a kernel code to iterate through data along the row direction, one can use the shared memory to enable memory coalescing. The technique is illustrated in Figure 6.10 for matrix multiplication. Each thread reads a row from d_M, a pattern that cannot be coalesced. Fortunately, a tiled algorithm can be used to enable coalescing. As we discussed in Chapter 5, threads of a block can first cooperatively load the tiles into the shared memory. Care must be taken to ensure that these tiles are loaded in a coalesced pattern. Once the data is in shared memory, it can be accessed either on a row basis or a column basis with much less performance variation because the shared memories are implemented as intrinsically high-speed, on-chip memory that does not require coalescing to achieve a high data access rate.

我们将图 5.7复制为图 6.11，其中矩阵乘法内核将两个矩阵d_M和d_N加载到共享内存中。请注意，线程块中的每个线程负责将一个d_M元素和一个d_N元素加载到每个阶段中的Mds和Nds中，如第 8 行中的for循环所定义。回想一下，每个图块中涉及TILE_WIDTH ^{2 个}线程。线程使用threadIdx.y和threadIdx.y来确定要加载的每个矩阵的元素。

We replicate Figure 5.7 here as Figure 6.11, where the matrix multiplication kernel loads two tiles of matrix d_M and d_N into the shared memory. Note that each thread in a thread block is responsible for loading one d_M element and one d_N element into Mds and Nds in each phase as defined by the for loop in line 8. Recall that there are TILE_WIDTH² threads involved in each tile. The threads use threadIdx.y and threadIdx.y to determine the element of each matrix to load.

d_M元素在第 9 行加载，其中每个线程的索引计算使用 m 来定位图块的左端。然后，图块的每一行都由TILE_WIDTH线程加载，这些线程的threadIdx在x维度上不同。由于这些线程具有连续的threadIdx.x值，因此它们位于相同的扭曲。此外，索引计算d_M[row][m*TILE_SIZE+tx]使这些线程访问同一行中的元素。问题是扭曲中的相邻线程是否确实访问了行中的相邻元素。回想一下，同一行中的元素被放置到全局内存的连续位置中。由于列索引m*TILE_SIZE+tx使得所有具有相邻tx值的线程都将访问相邻的行元素，所以答案是肯定的。硬件检测到同一 warp 中的这些线程访问全局内存中的连续位置，并将它们组合成合并访问。

The d_M elements are loaded in line 9, where the index calculation for each thread uses m to locate the left end of the tile. Each row of the tile is then loaded by TILE_WIDTH threads of which the threadIdx differ in the x dimension. Since these threads have consecutive threadIdx.x values, they are in the same warp. Also, the index calculation d_M[row][m∗TILE_SIZE+tx] makes these threads access elements in the same row. The question is whether adjacent threads in the warp indeed access adjacent elements in the row. Recall that elements in the same row are placed into consecutive locations of the global memory. Since the column index m∗TILE_SIZE+tx is such that all threads with adjacent tx values will access adjacent row elements, the answer is yes. The hardware detects that these threads in the same warp access consecutive locations in the global memory and combines them into a coalesced access.

在d_N的情况下，行索引m*TILE_SIZE+ty对于同一 warp 中的所有线程具有相同的值；它们都具有相同的ty值。因此，同一 warp 中的线程访问同一行。问题是扭曲中的相邻线程是否访问行的相邻元素。请注意，每个线程Col的列索引计算基于bx*TILE_SIZE+tx（参见第 4 行）。因此，线程束中的相邻线程访问行中的相邻元素。硬件检测到同一 warp 中的这些线程访问全局内存中的连续位置，并将它们组合成合并访问。

In the case of d_N, the row index m∗TILE_SIZE+ty has the same value for all threads in the same warp; they all have the same ty value. Thus, threads in the same warp access the same row. The question is whether the adjacent threads in a warp access adjacent elements of a row. Note that the column index calculation for each thread Col is based on bx∗TILE_SIZE+tx (see line 4). Therefore, adjacent threads in a warp access adjacent elements in a row. The hardware detects that these threads in the same warp access consecutive locations in the global memory and combine them into a coalesced access.

读者会发现，基于图 6.11中的内核代码绘制一幅图并识别加载图块每个元素的线程的threadIdx.y和threadIdx.x值非常有用。图 6.11中的第 5、6、9 和 10 行构成了一种常用的编程模式，用于在平铺算法中将矩阵元素加载到共享内存中。我们还想鼓励读者通过第 12 行和第 13 行中的点积循环来分析数据访问模式。请注意，warp 中的线程不会访问Mds的连续位置。这不是问题，因为Mds位于共享内存中，不需要合并即可实现高速数据访问。

Readers shall find it useful to draw a picture based on the kernel code in Figure 6.11 and identify the threadIdx.y and threadIdx.x values of the thread that loads each element of the tile. Lines 5, 6, 9, and 10 in Figure 6.11 form a frequently used programming pattern for loading matrix elements into shared memory in tiled algorithms. We would also like to encourage readers to analyze the data access pattern by the dot-product loop in lines 12 and 13. Note that the threads in a warp do not access consecutive location of Mds. This is not a problem since Mds is in shared memory, which does not require coalescing to achieve high-speed data access.

6.3 执行资源的动态分区

6.3 Dynamic Partitioning of Execution Resources

流式多处理器 (SM) 中的执行资源包括寄存器、共享内存、线程块槽和线程槽。这些资源被动态分区并分配给线程以支持它们的执行。在第 4 章中，我们看到当前一代设备有 1,536 个线程槽，每个槽可容纳一个线程。这些线程槽在运行时被分区并分配给线程块。如果每个线程块由 512 个线程组成，则 1,536 个线程槽被划分并分配给三个块。在这种情况下，由于线程槽位的限制，每个SM最多可以容纳3个线程块。如果每个线程块包含128个线程，则将1,536个线程槽划分并分配给12个线程块。在线程块之间动态划分线程槽的能力使得 SM 具有多种用途。它们可以执行多个线程块，每个线程块具有几个线程，或者执行几个线程块，每个线程块具有多个线程。这与固定分区方法相反，在固定分区方法中，每个块接收固定数量的资源，而不管其实际需求如何。当块具有很少的线程并且无法支持需要比固定分区允许的更多线程槽的块时，固定分区会导致线程槽的浪费。

The execution resources in a streaming multiprocessor (SM) include registers, shared memory, thread block slots, and thread slots. These resources are dynamically partitioned and assigned to threads to support their execution. In Chapter 4, we have seen that the current generation of devices have 1,536 thread slots, each of which can accommodate one thread. These thread slots are partitioned and assigned to thread blocks during runtime. If each thread block consists of 512 threads, the 1,536 thread slots are partitioned and assigned to three blocks. In this case, each SM can accommodate up to three thread blocks due to limitations on thread slots. If each thread block contains 128 threads, the 1,536 thread slots are partitioned and assigned to 12 thread blocks. The ability to dynamically partition the thread slots among thread blocks makes SMs versatile. They can either execute many thread blocks each having few threads, or execute few thread blocks each having many threads. This is in contrast to a fixed partitioning method where each block receives a fixed amount of resources regardless of their real needs. Fixed partitioning results in wasted thread slots when a block has few threads and fails to support blocks that require more thread slots than the fixed partition allows.

资源的动态分区可能会导致资源限制之间微妙的相互作用，从而导致资源利用不足。这种交互可以发生在块槽和线程槽之间。例如，如果每个块有128个线程，则可以将1,536个线程槽划分并分配给12个块。然而，由于每个 SM 中只有 8 个块槽，因此只允许使用 8 个块。这意味着仅使用 1,024 个线程槽。因此，为了充分利用块槽和线程槽，每个块中至少需要256个线程。

Dynamic partitioning of resources can lead to subtle interactions between resource limitations, which can cause underutilization of resources. Such interactions can occur between block slots and thread slots. For example, if each block has 128 threads, the 1,536 thread slots can be partitioned and assigned to 12 blocks. However, since there are only 8 block slots in each SM, only 8 blocks will be allowed. This means that only 1,024 of the thread slots will be utilized. Therefore, to fully utilize both the block slots and thread slots, one needs at least 256 threads in each block.

正如我们在第 4 章中提到的，CUDA 内核中声明的自动变量被放入寄存器中。一些内核可能使用大量自动变量，而另一些内核可能使用很少的自动变量。因此，我们应该预料到一些内核需要很多寄存器，而另一些则需要更少。通过在块之间动态划分寄存器，如果需要很少的寄存器，SM 可以容纳更多的块；如果需要更多的寄存器，则可以容纳更少的块。然而，人们确实需要意识到寄存器限制和其他资源限制之间的潜在相互作用。

As we mentioned in Chapter 4, the automatic variables declared in a CUDA kernel are placed into registers. Some kernels may use lots of automatic variables and others may use few of them. Thus, one should expect that some kernels require many registers and some require fewer. By dynamically partitioning the registers among blocks, the SM can accommodate more blocks if they require few registers and fewer blocks if they require more registers. One does, however, need to be aware of potential interactions between register limitations and other resource limitations.

在矩阵乘法示例中，假设每个 SM 有 16,384 个寄存器，并且内核代码每个线程使用 10 个寄存器。如果我们有 16×16 个线程块，那么每个 SM 上可以运行多少个线程？我们可以通过首先计算每个块所需的寄存器数量来回答这个问题，即10×16×16=2,560。六个块所需的寄存器数量为 15,360，低于 16,384 的限制。添加另一个块将需要 17,920 个寄存器，这超出了限制。因此，寄存器限制允许在每个 SM 上运行总共有 1,536 个线程的块，这也符合块槽和 1,536 个线程槽的限制。

In the matrix multiplication example, assume that each SM has 16,384 registers and the kernel code uses 10 registers per thread. If we have 16×16 thread blocks, how many threads can run on each SM? We can answer this question by first calculating the number of registers needed for each block, which is 10×16×16=2,560. The number of registers required by six blocks is 15,360, which is under the 16,384 limit. Adding another block would require 17,920 registers, which exceeds the limit. Therefore, the register limitation allows blocks that altogether have 1,536 threads to run on each SM, which also fits within the limit of block slots and 1,536 thread slots.

现在假设程序员在内核中声明另外两个自动变量，并将每个线程使用的寄存器数量增加到 12 个。假设相同的 16×16 块，每个块现在需要12×16×16=3072个寄存器。 6 个块所需的寄存器数量现在为 18,432 个，超出了寄存器限制。 CUDA运行时系统通过将分配给每个SM的块数量减少1来处理这种情况，从而将所需的注册数量减少到15,360个。但是，这会将 SM 上运行的线程数从 1,536 减少到 1,280。也就是说，通过使用两个额外的自动变量，程序发现每个 SM 中的扭曲并行度减少了六分之一。这有时被称为“性能悬崖”，其中资源使用量的轻微增加可能会导致并行性和性能的显着降低[RRS2008]。读者可以参考 CUDA 占用计算器[NVIDIA]，它是一个可下载的 Excel 表格，根据内核对资源的使用情况，计算特定设备实现的每个 SM 上运行的实际线程数。

Now assume that the programmer declares another two automatic variables in the kernel and bumps the number of registers used by each thread to 12. Assuming the same 16×16 blocks, each block now requires 12×16×16=3,072 registers. The number of registers required by six blocks is now 18,432, which exceeds the register limitation. The CUDA runtime system deals with this situation by reducing the number of blocks assigned to each SM by one, thus reducing the number of registered required to 15,360. This, however, reduces the number of threads running on an SM from 1,536 to 1,280. That is, by using two extra automatic variables, the program saw a one-sixth reduction in the warp parallelism in each SM. This is sometimes a referred to as a “performance cliff” where a slight increase in resource usage can result in significant reduction in parallelism and performance achieved [RRS2008]. Readers are referred to the CUDA Occupancy Calculator [NVIDIA], which is a downloadable Excel sheet that calculates the actual number of threads running on each SM for a particular device implementation given the usage of resources by a kernel.

6.4 指令混合和线程粒度

6.4 Instruction Mix and Thread Granularity

性能调优中一个重要的算法决策是线程的粒度。在每个线程中投入更多工作并使用更少的线程通常是有利的。当线程之间存在一些冗余工作时，就会出现这种优势。在当前一代的设备中，每个 SM 的指令处理带宽有限。每条指令都会消耗指令处理带宽，无论是浮点计算指令、加载指令还是分支指令。消除冗余指令可以缓解指令处理带宽的压力，提高内核的整体执行速度。

An important algorithmic decision in performance tuning is the granularity of threads. It is often advantageous to put more work into each thread and use fewer threads. Such advantage arises when some redundant work exists between threads. In the current generation of devices, each SM has limited instruction processing bandwidth. Every instruction consumes instruction processing bandwidth, whether it is a floating-point calculation instruction, a load instruction, or a branch instruction. Eliminating redundant instructions can ease the pressure on the instruction processing bandwidth and improve the overall execution speed of the kernel.

图 6.12说明了矩阵乘法中的这种机会。图 6.11中的平铺算法使用一个线程来计算输出d_P矩阵的一个元素。这需要d_M的一行和d_N的一列之间的点积。

Figure 6.12 illustrates such an opportunity in matrix multiplication. The tiled algorithm in Figure 6.11 uses one thread to compute one element of the output d_P matrix. This requires a dot product between one row of d_M and one column of d_N.

线程粒度调整的机会来自于多个块冗余地加载每个d_M瓦片的事实。如图6.12所示，相邻图块中的两个d_P元素的计算使用相同的d_M行。使用原始分片算法，相同的d_M行由分配的两个块冗余加载以生成这两个Pd分片。可以通过将两个线程块合并为一个来消除这种冗余。新线程块中的每个线程现在计算两个d_P元素。这是通过修改内核来完成的，以便由内核的最内层循环计算两个点积。两个点积都使用Mds行相同但Nds列不同。这将全局内存访问减少了四分之一。鼓励读者编写新内核作为练习。

The opportunity of thread granularity adjustment comes from the fact that multiple blocks redundantly load each d_M tile. As shown in Figure 6.12, the calculation of two d_P elements in adjacent tiles uses the same d_M row. With the original tiled algorithm, the same d_M row is redundantly loaded by the two blocks assigned to generate these two Pd tiles. One can eliminate this redundancy by merging the two thread blocks into one. Each thread in the new thread block now calculates two d_P elements. This is done by revising the kernel so that two dot products are computed by the innermost loop of the kernel. Both dot products use the same Mds row but different Nds columns. This reduces the global memory access by one-quarter. Readers are encouraged to write the new kernel as an exercise.

潜在的缺点是新内核现在使用更多的寄存器和共享内存。正如我们在上一节中讨论的，每个 SM 上可以运行的块数量可能会减少。它还将线程块的总数减少了一半，这可能会导致较小维度的矩阵的并行度不足。在实践中，我们发现组合最多四个相邻水平块来计算相邻水平图块可以提高大型（2,048×2,048 或更多）矩阵乘法的性能。

The potential downside is that the new kernel now uses even more registers and shared memory. As we discussed in the previous section, the number of blocks that can be running on each SM may decrease. It also reduces the total number of thread blocks by half, which may result in an insufficient amount of parallelism for matrices of smaller dimensions. In practice, we found that combining up to four adjacent horizontal blocks to compute adjacent horizontal tiles improves the performance of large (2,048×2,048 or more) matrix multiplication.

6.5 总结

6.5 Summary

在本章中，我们回顾了 CUDA 设备上 CUDA C 应用程序性能的主要方面：控制流发散、全局内存合并、动态资源分区和指令混合。我们提出了为这些性能方面创建良好程序模式的实用技术。我们将在接下来的几章中继续在案例研究中研究这些技术的实际应用。

In this chapter, we reviewed the major aspects of CUDA C application performance on a CUDA device: control flow divergence, global memory coalescing, dynamic resource partitioning, and instruction mixes. We presented practical techniques for creating good program patterns for these performance aspects. We will continue to study practical applications of these techniques in the case studies in the next few chapters.

6.6 练习

6.6 Exercises